NY – Rachel S lives in a walkable neighborhood in Brooklyn, New York. Most days she can live comfortably without a car. She often works remotely, but occasionally needs to go to the office. That's where her situation becomes a little challenging. Her workspace is not easily accessible by public transportation.
Since he doesn't need to drive frequently, he applied for the Zipcar car-sharing platform to meet his occasional needs. The application process is fairly quick, allowing consumers to hit the road using their fleet of cars relatively quickly.
Unfortunately, that wasn't the case for Rachel. As soon as she hit the send button, the artificial intelligence software the company uses deemed her ineligible. Baffled by the result, Rachel contacted the company's customer service team.
After all, she has no demerits that suggest she is an irresponsible driver. She has no points on her license. Her only failure was a traffic ticket she received when she was seventeen and the fine was paid years ago.
Although the traffic ticket has since been rectified, now, at thirty years old, he is still dealing with the consequences.
Spoke to Zipcar's customer service team to no avail. Despite having a clean driving record, she was turned away. She claims the company said it had no recourse and that a human couldn't override the decision.
“There was no path or process to appeal to a human being and, while it is reasonable, the only way to try again would be to reapply,” for which there is a non-refundable application fee, Rachel told Al Jazeera remembering her conversation with the company.
Zipcar did not respond to Al Jazeera's request for comment.
Rachel is one of many consumers who have had loans, memberships, and even job opportunities turned down by AI systems without any recourse or appeals policy, as businesses continue to rely on AI to make key decisions that impact everyday lives.
That includes D, who recently lost his job.
As a condition of the interview, D requested that we only use his initial out of respect for his privacy. D religiously searched for a new opportunity without success.
After months of searching, D finally landed a job, but there was one big problem: timing.
It was still several weeks before D would start the new job and several weeks after D would receive his first paycheck.
For additional help, D applied for a personal loan on multiple platforms in an effort to bypass predatory payday loans, just to get by in the meantime.
D was rejected for all the loans they applied for. Although D did not confirm which specific companies, the sector has multiple options, including Upstart, Upgrade, SoFi, Best Egg, and Happy Money, among others.
D says that when they called the companies after submitting an online application, no one could help and there were no appeals.
When D was in his early twenties, he had a credit card that he couldn't pay his bills with. That was his only credit card. They also rent an apartment and rely on public transportation.
According to AI-powered online lenders, their lack of credit history and collateral makes them ineligible for a loan despite having paid off their outstanding debt six years ago.
D did not confirm which specific companies they attempted to obtain a loan from. Al Jazeera approached each of those companies to comment on their processes; only two responded, Upgrade and Upstart, at the time of publication.
“There are cases where we can change the loan decision based on additional information, i.e. proof of other sources of income, that was not provided in the original application, but when it comes to a 'human decision,' “There are a lot of room for personal bias, something that regulators and industry leaders have worked hard to eliminate,” a spokesperson for the Upgrade company said in an email to Al Jazeera. “Technology has brought objectivity and fairness to the lending process , and now decisions are made based on the true merit of the applicant.”
Amplified historical biases
But it's not as simple as that. Existing historical biases are often amplified by modern technology. According to 2021 research by The Markup, Black Americans are 80 percent more likely to be automatically rejected by lending agencies than their white counterparts.
“AI is just a model that is trained on historical data,” said Naeem Siddiqi, senior advisor at SAS, a global data and artificial intelligence company, where he advises banks on credit risk.
This is due to the long history of discriminatory practices in banking towards communities of color in the United States.
“If you take biased data, the only thing the AI or any model will do is essentially repeat what you gave it,” Siddiqui said.
“The system is designed to make as many decisions as possible with as little bias and human judgment as possible so that it is an objective decision. This is the irony of the situation… of course, there are some who fall through the cracks,” Siddiqi added.
It is not just a question of race. Companies like Apple and Goldman Sachs have even been accused of systematically granting lower credit limits to women than men.
These concerns are also generational. Siddiqi says such denials also overwhelmingly limit social mobility among younger generations, such as young millennials (those born between 1981 and 1996) and Generation Z (those born between 1997 and 2012), across all demographic groups.
This is because the standard moniker of good financial health – including credit cards, houses and cars – when evaluating someone's financial responsibility is becoming less and less relevant. Only about half of Generation Z have credit cards. That's a decrease from all previous generations.
Members of Generation Z are also less likely to have collateral, such as a car, to gamble on when applying for a loan. According to a recent McKinsey study, this age group is less likely to choose to obtain a driver's license than previous generations. Only a quarter of 16-year-olds and 45 percent of 17-year-olds have a driving license. That's down 18 percent and 17 percent, respectively.
The Consumer Financial Protection Bureau has stepped up its safeguards for consumers. In September, the agency announced that credit bureaus will now have to explain the reason behind denying a loan.
“Creditors often feed these complex algorithms with large data sets, sometimes including data that can be gleaned from consumer surveillance. As a result, a consumer may be denied credit for reasons that he or she may not consider particularly relevant to his or her finances,” the agency said in a statement.
However, the agency does not address the lack of a humane appeals process, as D claims to have dealt with it personally.
D said they had to postpone paying some bills that will hurt their long-term financial health and could affect their ability to get a loan with reasonable interest rates, if at all, in the future.
'Excluded from opportunities'
Siddiqi suggests that lenders should start considering alternative data when making a lending decision, which can include rent and utility payments and even social media behavior as well as spending patterns.
On social media, foreigner registrations are a key indicator.
“If you have more money, you tend to travel more or if you follow sites like Bloomberg, Financial Times and Reuters, you are more likely to be financially responsible,” adds Siddiqi.
The problem of automatic rejection is not only a problem for loan and membership applications, but also for job opportunities. On social media platforms like Reddit, users post rejection emails that they receive immediately after submitting a request.
“I met all the requirements and hit all the keywords and within a minute of submitting my application, I received both the application acknowledgment and the rejection letter,” Matthew Mullen, the original poster, told Al Jazeera.
The Connecticut-based video editor says this was a first for him. Experts like Lakia Elam, director of HR consultancy Magnificent Differences Consulting, say that between applicant tracking systems and other AI-powered tools, this is becoming an increasingly important and increasingly problematic issue.
Applicant tracking systems often overlook transferable skills that don't always align on paper with a candidate's skill set.
“Often, applicants who have a non-linear career path, many of whom come from diverse backgrounds, are excluded from opportunities,” Elam told Al Jazeera.
“I keep telling organizations that we need to keep the human touch in this process,” Elam said.
But more and more organizations are relying more on programs like ATS and ChatGPT. Elam maintains that leaves out many worthwhile job seekers, including herself.
“If I were to go through an AI system today, I guarantee you I would be rejected,” Elam said.
You have a GED (the equivalent of a high school diploma) instead of a four-year degree.
“They see GED on my resume and say we should stay away from this,” Elam added.
That's partly why Americans don't want AI involved in the hiring process. According to an April 2023 report from Pew Research, 41 percent of Americans believe AI should not be used to review job applications.
“It's part of a larger conversation about losing our way to due process,” Rachel said.
