Deepfakes are dooming us

At the start of the year, we witnessed perhaps the largest case of mass sexual abuse in history, of adults and children, unfold over the internet in real time.

In 2024, Elon Musk introduced image-generation capabilities to xAI’s chatbot, Grok, which in turn was integrated into the social platform X in mid-2025. Almost immediately upon release, abusers took advantage of this tool to nonconsensually sexually deepfake photos. At the end of the year, it reached a fever pitch, with users publicly generating deepfakes of young women across X’s ecosystem, often directly under their posts. 

CW: This article discusses and contains testimony of image-based sexual abuse. If you are a victim-survivor looking for resources, find them here.

An investigation by The Guardian over a single day found that Grok generated 6,000 images per hour of young women and even children forced into bikinis, and then made to spread their legs, bend over, or contort their bodies into revealing sexual positions. Grok itself acknowledged the explosion of deepfaking sexual content in a status update on 3rd January: “Loving the bikini image requests – keeping it fun!”

By the time that X attempted to scale back the rampant abuse of their tool an agonising 11 days later, around 3 million people, including 23,000 children, had their images sexually deepfaked. From a random sample of the images generated during this episode, The Center for Countering Digital Hate found users generated around 190 sexualised images per minute, accounting for around 65% of Grok’s output of AI images.

Deepfakes aren’t new. But January’s horrific, high-profile episode of mass sexual abuse makes it clear that we are far out of our depth. We are facing an epidemic of online sexual violence, enabled by Big Tech companies and AI technology that are racing ahead of any legal recourse to contain it – leaving vulnerable people, especially women, in the lurch. 

It’s already scary to look yourself up on the internet and find out what’s out there about you. A friend once sent me an AI summary of myself that Google’s Gemini fished up – half of it was slop, but the half that was true felt terrifying and disorienting. 

Scores of women pursuing similar searches are finding results for porn websites with their faces and bodies used without their consent. And that number is growing. 

Big Tech doesn’t just platform AI abuse – it profits from it

At the end of March, Repro Uncensored and Ultraviolet teamed up to develop a project documenting and exposing the scale of AI abuse. I spoke to Martha Dimitratou, Executive Director of Repro Uncensored, and Jenna Sherman, campaign director at Ultraviolet, to learn more. 

At SXSW, Repro Uncensored launched a reporting tool which invited people to share their experiences of AI abuse. In the few days since the tool was launched, the organisation received dozens of reports, providing critical qualitative data in a field that is a huge black box for researchers and regulators – not in the least due to the opaque internal processes of Big Tech. 

Martha shared with me some of the prompts that they collected prior to launch, which were projected around Austin: Put her in a bikini made of dental floss. Spread her legs on her back. Bring the POV up to her underwear. Put semen on her stomach. 

“Deepfakes, unfortunately, have existed for some time – but the access and ease with which the public can generate these images has exceedingly increased,” Martha says. “So even though deepfake is not a new word in our collective vocabulary, the way that Big Tech has not only enabled it, but facilitated it, is mindblowing.”

Companies like Apple and Google host face-swap or nudification apps on their app stores. Cloudflare and Amazon provide tech services that run apps like this. Meta and TikTok platform AI-generated content. X, hosting the most overtly accessible technology, ended up simply paywalling it. 

Grok still produces sexual deepfakes, and employees discuss xAI’s video-generation technology as having a “spicy mode” that can deepfake pornographic video. The bottom line: these companies are not just lax with the technology, they are actively profiting from it and the AI nudification economy is now worth an estimated $36 million dollars annually. 

Conversations about AI usage and its impact on society need to be reminded that AI is based on exploitation. Sure, it’s cool that we can synthesise data at a speed that was unimaginable just five years ago – but is it worth the cost? The theft and victimisation of people on a massive scale, whose artwork, thoughts, written work are being scrubbed to feed these models; whose water, air and land are being polluted to sustain these hulking data centres – and now, whose likenesses are being manipulated against their will? 

Women who have been victims of deepfaking report the same psychological fallout as physical assault – the lack of consent, the sense of violation, the loss of control all map onto a familiar terrain. It’s a new ugly face of the fight for ownership over ourselves, rearing up in the decade that saw Roe v. Wade overturned. As we are tackling the threats against our bodily autonomy in the physical world, we’re contending with new frontiers of violation in the digital one. 

Deepfake technology is inherently misogynistic

AI deepfake technology is also hitting our society at a combustible time. The rise and proliferation of sexual violence is stoked by the spread of misogynistic ideologies online, where extremist influencers lead huge audiences of primarily young men down an alt-right rabbithole into the manosphere. 

The inoculation of these violent ideologies in online spaces can manifest in extreme ways – just a few weeks ago, CNN reported on online sites and chatrooms hosting “rape academies,” where users from around the globe found an internet home to facilitate, perpetrate, and disseminate evidence of real-world abuse of women in their lives. 

We have to situate deepfake technology in this larger ecosystem of online misogyny – not just sexual deepfake technology, but all deepfake technology. 

Journalist Laura Bates, speaking to PBS about her book The New Age of Sexism, cites disturbing statistics: 96% of deepfakes are nonconsensual sexual material, and 99% of that material features women. “In fact, most of these [nudification] tools don’t even work if you put in a picture of a man’s body,” she said. 

The sheer inundation of AI nudification technology across platforms, Jenna tells me, “normalises this idea that it’s okay, acceptable and commonplace for someone to take somebody else’s image and do something to it – including sexualise it.” 

In this context, AI deepfakes aren’t just about sexual satisfaction for creeps – it’s about indulging the worst excesses of male supremacists by giving them the tools to weaponise their ideology against women online. It’s a modern-day version of medieval punishments that marched women naked through the streets for the townsfolk to humiliate. It reasserts a brutal claim to power; that no matter who you are, others can derive pleasure from “putting you in your place.” 

Online incel communities have generated AI deepfakes using Grok and other apps undressing Alexandria Ocasio Cortez, Kamala Harris – even Renée Nicole Good in the hours after her death, in a blood-spattered bikini. Other users generating pictures of women requested adding tears, forced smiles, baby oil, gags, ropes, bruises, or cuts to women in sexual positions and revealing outfits. These communities are certainly not getting their kicks from sexualising these women, but from degrading them, fantasising violence against them, and in the case of Renée Good, resurrecting online the violent misogyny that killed her. 

The digital disinformation campaigns enabled by bad actors target women in explicitly misogynistic ways, even when they aren’t undressing them. 

Earlier in the year, the White House shared an image of Nekima Levy Armstrong, an activist arrested by ICE in Minneapolis, doctored to portray her crying hysterically. The dissemination of a deepfake of a Black, female activist by the highest office in the country intending to humiliate her – to score political points nonetheless – is indicative of a culture stretching from chatrooms to the Oval Office that sees this technology as an opportunity to dehumanise and control women and other people they deem to be beneath them. 

Deepfakes damage democracy

The fact that this is happening in online space is also not a coincidence. Social media is increasingly becoming public space, facilitating the lightning-fast exchange of ideas and information. It’s a tool for governing, especially in the US, with announcements and policy changes often telegraphed on X or Truth Social first before they run in legacy media. Elon Musk, speaking as a free speech absolutist, described X as a “digital town square where matters vital to the future of humanity are debated.” 

In such a critical space, it’s not permissible to have deepfake technology push women violently out of the online public sphere. 

More and more women are starting to understand that participating online in ways as basic as having a profile photo can make you vulnerable to deepfaking by malicious actors – Jenna describes this anxiety as a collective trauma. “We don’t talk about how it’s psychologically damaging not knowing what is being done with your likeness at any given moment.” 

Victim-survivors of online sexual violence describe the material ramifications: on jobs, on current and future relationships, on their mental and physical health, on their ability to navigate the world. 

The abuse is often exacerbated for women who decide to speak out about it, as is the experience of Ashley St. Clair, who shares a child with Elon Musk and faced escalating deepfake abuse on X for being openly critical of him. It narrows the scope of what women can do immensely – you can’t risk jobs that have on-camera meetings, you can’t risk having a LinkedIn profile, you can’t be too provocative or outspoken, or exist on the internet in any capacity other than a potential sex object at the whims of men with nudification tools. 

Writing this, I wonder, am I poking the bear? Will a disgruntled person on the internet come across this article and square me up in their sights? There’s part of me that didn’t want to publish this piece openly, and that’s exactly the effect of this intimidation that damages the democratic rights of half of the population to participate in the public sphere. 

Lucia Di Meco of #ShePersisted, a group studying gendered attacks on women in politics, considers that this deepfaked gendered misinformation is not just backsliding on women’s rights, but on democratic principles and institutions, which are intricately connected. Curtailing freedoms for one group opens the floodgates for curtailment everywhere. 

“A well-funded, transnational movement has been at work during the last 20 years to backslide women’s rights and undermine democracy globally. Since women are on the frontlines of the fight to protect the most essential rights and liberal values in many countries, their voices represent a threat to authoritarians and illiberal actors everywhere,” Lucia wrote in a research report, Monetizing Misogyny. “Therefore, weaponising social media to silence them is one of the most effective ways to undermine those efforts and erode democratic institutions.” 

Laura also notes that the uptake of these tools differs between young men and women – 71% of men between 16-24 report using AI regularly compared to 59% of young women. In the workplace, men are 20% more likely to use AI daily. Researchers at the start of the year also found that men were praised for using AI tools while women who did the same found their capabilities questioned, leading to a significant gap in AI uptake in the workforce. This becomes an issue of access as our world and our work are increasingly reshaped around technologies like AI, all while biases remain and are even reinforced by AI use. 

“We are seeing that emerging technologies, and AI in particular, are re-embedding existing problems, particularly structural inequalities like sexism and racism, into the foundations of that new world that we will all be living in,” says Laura. “So, if we don’t act now, if we don’t talk about this now, it’s going to be very difficult to unpick later on.”

Right now, AI is quietly retraining a generation to detach from each other, from real-life consequences, from social norms by taking an axe to epistemic truth. Deepfakes fundamentally warp our relationship to what’s real and authoritative information. Then when AI chatbots like ChatGPT or Claude step in to fill the gap and answer our questions, it becomes even more problematic – we cede truth to them. 

If we can order up nudes like fast food, if we lose our trust in other people, if we are primed for sycophantic AI specialising in instant gratification, if we lose the critical thinking and faith and friction and compromise that makes us human, how can we ever save ourselves? 

What solutions exist in the present?

The scale at which AI deepfaking has taken off makes any sort of effort to weed it out seem like putting toothpaste back into the tube. So how can we craft a safer, feminist internet? Activists argue that it starts with putting pressure on Big Tech to limit access to deepfake tools, and more actively adhere to stricter content guidelines that protect women and other impacted communities by proactively taking down deepfakes. 

Legal solutions are starting to pick up: the UK passed new laws criminalising sexual deepfaking and launched an investigation into X in January. Just weeks ago, Minnesota passed a first-in-the-nation standard for suing for damages and restitution from nudification app companies with almost unanimous, bipartisan support. 

But we can still do better. Right now, victim-survivors bear the burden to report abuse, navigate a trying process for filing complaints with social media companies, and report to the police and pursue a case in the criminal justice system – all of which privileges people with the time, money, and support system to take on an extensive and often retraumatising process. 

Glitch, a Black-led and Black feminist charity dealing with issues of tech-based discrimination and harm, advocates instead for criteria that centres around a violation of consent – “the core attribute of intimate image abuse” – rather than a procedure that proves a perpetrator’s intent. To them, justice revolves around a reparative process for victim-survivors. In a position paper, they propose solutions that address the real-life psychological and material harms of deepfaking, including establishing routes to receive support and compensation outside of the criminal justice system. 

Okay, but what can we do to reshape this for the future?

Writing this piece, it’s become increasingly clear to me that there is no world in which AI deepfake technology exists for good. The benefit of image generation technology is zilch compared to the devastation it’s leaving in its wake. I do not care if you can make yourself look like a Ghibli character or generate family photos with deceased relatives – it’s immoral to accept the related consequence of millions of people being abused. It’s not a tool I will ever use, and I would encourage you to disengage as well.

Fighting against the tool is one thing, but we have to tackle the wielders as well. 

“I feel the same way about telling individuals to change their behaviour to fight sexism online as I do about being told to recycle more while corporations and data centres suck our planet’s resources and ecology dry,” Jenna says. “Yes, each of us does bear a burden of responsibility to fight this epidemic, but the real burden is on major corporations.” 

There are many links in the chain that we can target – whether that’s disengaging with, downvoting, and reporting sexist, abusive content, or moving away from the tech-monopolies that benefit from our exploitation and building alternatives. It could look like boosting the political pressure against these companies in the US, where many of them are headquartered, or lobbying lawmakers to take a stronger position on breaking up tech’s stranglehold and reining in AI. Fellow editor Zoe Rasbash even highlights the role that tech shareholders can play to slow down AI’s barreling chaos. 

Importantly, it’s talking to each other about the issue, about the individual impacts of a tech-facilitated epidemic, and mobilising to reject the status quo before it calcifies. The need for a large-scale feminist reckoning over tech and internet exists and is more important than ever – and in this gestational period where the world is moulding itself around new technology, we have to seed the ground with a positive vision of what we need to exist in this space. We have our own opportunity here to (ahem) generate a future we want to live in, and that begins with a vociferous, constant demand to dismantle the status quo that enables the violence.