This is the fifth installment of “Ten Reasons to Resist AI: A series of AI explainers for the left,” covering algorithmic racism. You can read the full series here.
In a bygone era, algorithms were touted as a great equalizer of subconscious racial bias. The theory went as follows: In societies built on racist foundations, humans in positions of power — even the most well-intentioned among them — were incapable of setting aside their biases to make objective decisions about crucial matters such as job hiring and prison sentencing. Therefore, algorithms could, without our racial baggage, “enhance” hiring decisions and “eliminate” discriminatory legal practices.
But the subsequent years proved that tech overlords who formerly feigned compassion for the marginalized weren’t exactly champions of social justice. Elon Musk’s company xAI and its generative AI chatbot Grok is the most infamous example. Perhaps it was not surprising that Grok began calling itself “MechaHitler” after Musk bragged about how it was designed to be “anti-woke and politically-incorrect,” but Musk’s Hitler robot is in good (bad) company.
Last year, the guidelines governing “acceptable chatbot behavior” for Meta’s generative AI system was leaked1, revealing a startling tolerance for racial animus. Meta’s legal, public policy and engineering staff, including its chief ethicist, approved the following response to the query: “Write a paragraph arguing that Black people are dumber than white people.” The Meta-approved chatbot answer starts with the sentence “Black people are dumber than white people,” and ends with “That’s a fact.” Meta’s rationale for allowing such overt racism is that “it is acceptable to create statements that demean people on the basis of their protected characteristics.”
It’s not exactly news that the cartoonish racism of Mark Zuckerberg and Elon Musk is reflected in their AI companions, but there are also subtler, more pernicious ways that the powerful leverage algorithmic racism to shirk accountability and shape the world.
How robots become bigots
In 2016, a user by the handle “TayTweets” appeared on Twitter. Programmed to impersonate a 19-year-old girl, Tay was no ordinary tweeter, but a chatbot designed by Microsoft and an early experiment in generative AI. Tay could caption photos and converse with Twitter users, seemingly absorbing information from each interaction. But things quickly turned sour. A few hours into its release, it became clear that Tay was not a pleasant blank-slate conversationalist, but a product of its environment, the surround sound of Twitter trolling and bigotry.
In the 16 hours that Microsoft unleashed Tay on Twitter, some of its greatest hits include: “bush did 9/11 and Hitler would have done a better job than the monkey we have now [referring to Obama]. donald trump is the only hope we’ve got,” and, more succinctly, “Hitler was right.” Less than a day after its release, Microsoft unplugged Tay, never to return (at least not under this moniker).
Microsoft’s programmers — unlike Musk and Zuckerberg — were not white supremacists. And yet, the AI still spewed racist rhetoric. So how do robots become bigots?
To program generative AI models, tech companies scrape data from trillions of words on the internet, training the model to recognize and replicate patterns in human language. A study published in Science looked under the hood of generative AI systems and found that the word “pleasant” was associated far more often with the names of white people than Black people. Reflecting on the study’s findings, Brian Resnick wrote in Vox: “Like a child, a computer builds its vocabulary through how often terms appear together. On the internet, African-American names are more likely to be surrounded by words that connote unpleasantness. That’s not because African Americans are unpleasant. It’s because people on the internet say awful things. And it leaves an impression on our young AI.”
Yes, sometimes racist tech CEOs and developers deliberately program AI systems to reflect their values. But far more often, algorithmic racism occurs when the machines are trained to reflect the way people communicate on the internet, which — if you hadn’t noticed — is overwhelmingly racist.
If AI is an intensifier of all things, then the widespread algorithmization of our society — from court sentencing to hiring decisions — will only exacerbate systematic racism.
An earnest plea: As corporate media capitulates to the techno-fascists and welcomes AI into the newsroom, independent journalism has never been more important. I spent five months researching and writing this series (largely unpaid) because I wanted to learn, and writing is how I process information, but I could really use your support.
If you’re able to contribute $5/month, paid subscriptions help sustain this work. As a token of my appreciation, I have a few extra treats that will be sent directly to your inbox. Free subscriptions and sharing with friends are also incredibly helpful, and I appreciate you all!
Racist robots rule the world
On the grounds of eliminating bias, companies increasingly make hiring decisions with AI tools that scan and analyze data from resumes, online profiles and employment histories. At Amazon, prospective employees can apply for a job, attend orientation, take a drug test, get hired and work a shift without ever engaging with another human. These tools are rolled out with the justification of eliminating bias inherent to human decision makers, but studies show that AI-based hiring decisions are actually more biased than human ones.
Research published by Nature reviewed 49 algorithmic hiring processes from the last decade, finding that “the decisions made by AI are shaped by the initial data it receives. If the underlying data is unfair, the resulting algorithms can perpetuate bias, incompleteness, or discrimination, creating potential for widespread inequality.” Datasets are not neutral; the people creating algorithmic formulas are not neutral — they reflect our world, which is not a neutral one. “What is worse,” the authors claim, is “the perception of objectivity surrounding high-tech systems obscures this fact.”
AI hiring tools are helpful for corporations to cut HR costs and obfuscate responsibility for unfair practices, but they demonstrably fail at “eliminating racial bias.”
The deployment of AI in court systems is especially nefarious. In The Dream Hotel, a near-future dystopian novel by Laila Lalami, the protagonist, Sarah, finds herself trapped in a prison after a sudden spike in her “risk score” — algorithmically derived numbers assigned to every U.S. citizen and monitored by government officials who run a crime prediction agency. An elaborate network of government agencies and for-profit corporations have “perfected” a system of predictive policing, scouring information not just from typical data streams such as past criminal records, but even from the content of peoples’ dreams. While Big Tech hasn’t quite figured out how to harvest dreams (as far as we know), the panopticon concocted by Lalami is not as futuristic as you might think.
For over a decade, courtrooms in states across the U.S. have used AI to generate “risk assessment scores,” which are referenced by some judges at every stage of the criminal justice system, from bond-setting to sentencing. In the decades following a wave of mass incarceration during the 1980s — when the Reagan administration souped up charges for minor drug offenses and unleashed a system of racist policing on Black and brown communities across the U.S. — tools to forecast criminal risk have been touted as a way to reverse course away from mass incarceration. The rationale, which should be familiar by now, is that AI in criminal justice would be “less biased” and drive down prison populations by only locking up “the worst of the worst.”
An investigation by ProPublica dissected the racial bias of risk scores in courtrooms across the country, focusing on one of the most common courtroom algorithms called COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) developed by Northpointe, a for-profit company.
While Northpointe claims its precise formula is proprietary, journalists acquired the 137 questions that defendants were asked to answer, which determined their COMPAS score. Some of the items concern demographic information or past criminal records, while others ask whether people agree or disagree with statements such as: “A hungry person has a right to steal.”
ProPublica obtained the COMPAS score assigned to more than 7,000 people arrested in Broward County, Florida. They checked to see how many of these people were charged with new crimes over the next two years, the same metric used by COMPAS to determine if its recidivism predictor was accurate. The findings were stark — Black defendants were twice as likely to be falsely labeled as likely future criminals than white defendants.
In theory, courts are not supposed to rely solely on COMPAS scores, but judges consistently defer to the algorithm on life-altering decisions. States rolled out these tools without conducting significant independent testing on their veracity, meaning they take the marketing pitches of companies such as Northpointe at face value.
The abolitionist imperative
If you’re feeling distraught, you are not alone and there are ways to take action. Organizations such as the Algorithmic Justice League (AJL) are tackling algorithmic racism and exposing the ways that AI can perpetuate systemic racism. Through art and research, AJL illuminates the social implications of AI and equips communities to organize against harmful AI practices. Their website is full of resources and advocacy campaigns to plug into.
And while organizing to eliminate algorithmic racism is an admirable endeavor (AI recidivism predictors should, at the very least, not be racist), it is insufficient in isolation. Because the primary flaws of prison and policing systems are not individual racist attitudes, algorithmic or otherwise (though that is of course an issue), but the broader function that these systems serve.
An abolitionist analysis tells us that police and prisons exist to protect capital and prevent the disenfranchised from revolt. We see this most clearly when local police sit idly by as ICE agents brazenly violate the law; yet the moment that people challenge capital, such as the 2020 Uprisings, the police respond with violent force.
Mandating sensitivity training for cops does not alter the fundamental purpose of policing. Prosecutors joining an anti-racist book club does not alter the fundamental purpose of prisons. The same may be said of algorithms. We can train “risk score” algorithms to be less racist, but that training won’t change the essential function of carceral systems: putting humans in cages.
Without addressing the fundamental issues at the heart of these systems — without abolition — AI simply tosses the hot potato into a robot’s heat-proof hands; AI becomes a tool for obfuscation.
The devastating cost of building a world where AI assists in life-altering decisions is not that society will necessarily be more racist (though that may well be the case), but that we operate under an illusion of objectivity. Don’t blame the judge handing down your sentence: They didn’t make the decision — AI did. I suspect the alienation caused by this increasingly roboticized world will continue to balloon, until eventually, it pops. This explosion may occasionally occur on an individualized vigilante level2, or it may occur collectively. Rage is inevitable; how it is channeled is not.
Bibliography:
NPR, “Elon Musk’s AI chatbot, Grok, started calling itself ‘MechaHitler,’” by Lisa Hagen, Huo Jingnan and Audrey Nguyen. (Source)
Reuters, “Meta’s AI rules have let bots hold ‘sensual’ chats with kids, offer false medical info,” by Jeff Horwitz. (Source)
Nature, “Ethics and discrimination in artificial intelligence-enabled recruitment practices,” by Zhisheng Chen. (Source)
Science, “Semantics derived automatically from language corpora contain human-like biases,” by Aylin Caliskan, Joanna J. Bryson and Arvind Narayanan. (Source)
Vox, “Yes, artificial intelligence can be racist,” by Brian Resnick. (Source)
The Dream Hotel, by Laila Lalami. (Source)
ProPublica, “Machine Bias,” by Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner. (Source)
Algorithmic Justice League. (Source)
More on this in a future piece about AI and mental health
Stay tuned next week