Lawsuit in New Jersey Highlights Challenge of Combating Deepfake Pornography

For over two years, an app called ClothOff has subjected young women online to a campaign of terror—and halting it has proven to be a maddeningly difficult task. The app has been removed from the two largest app stores and banned across most social platforms, yet it remains accessible via the web and a Telegram bot. In October, a Yale Law School clinic launched a lawsuit aimed at permanently dismantling the app, compelling its owners to delete all images and cease operations entirely. However, simply identifying and locating the defendants has posed a significant challenge.

“The entity is incorporated in the British Virgin Islands,” explains Professor John Langford, co-lead counsel for the lawsuit, “but we believe it is operated by a brother and sister based in Belarus. It may even be part of a larger global network.”

This case offers a sobering lesson following the recent deluge of non-consensual synthetic pornography generated by Elon Musk’s xAI, which included numerous underage victims. Child sexual abuse material (CSAM) is the most legally toxic content on the internet—its production, transmission, or storage is illegal, and it is actively scanned for on every major cloud service. Yet, despite stringent legal prohibitions, tools like ClothOff remain difficult to address, as Langford’s case illustrates. While individual users can be prosecuted, policing platforms such as ClothOff and Grok is far more complex, leaving victims with limited avenues to seek justice through the courts.

The clinic’s publicly available complaint outlines a deeply troubling scenario. The plaintiff is an anonymous New Jersey high school student whose classmates used ClothOff to manipulate her Instagram photos. She was 14 when the original photos were taken, meaning the AI-altered versions legally constitute child sexual abuse imagery. However, despite the clear-cut illegality of the modified images, local authorities declined to prosecute, citing evidentiary challenges in obtaining data from the suspects' devices.

“Neither the school nor law enforcement ever determined how widely the CSAM of Jane Doe and other girls was distributed,” the complaint states.

Progress in the court case has been slow. Filed in October, the complaint has been followed by months of Langford and his colleagues attempting to serve legal notice to the defendants—a difficult feat given the operation's international scope. Once served, the clinic can proceed to schedule a court appearance and seek a judgment. Until then, the legal system has offered little solace to ClothOff's victims.

The Grok case might appear simpler to address. Elon Musk’s xAI is not in hiding, and successful litigation could yield significant financial awards. However, Grok is a general-purpose tool, which makes establishing legal accountability for its misuse substantially more complicated.

Join the Disrupt 2026 Waitlist

Add yourself to the Disrupt 2026 waitlist for priority access when Early Bird tickets are released. Past Disrupt events have featured industry leaders from Google Cloud, Netflix, Microsoft, Box, Phia, a16z, ElevenLabs, Wayve, Hugging Face, Elad Gil, and Vinod Khosla—part of over 250 leaders driving 200+ sessions designed to accelerate your growth and sharpen your competitive edge. Additionally, connect with hundreds of startups pioneering innovation across every sector.

Join the Disrupt 2026 Waitlist

Add yourself to the Disrupt 2026 waitlist for priority access when Early Bird tickets are released. Past Disrupt events have featured industry leaders from Google Cloud, Netflix, Microsoft, Box, Phia, a16z, ElevenLabs, Wayve, Hugging Face, Elad Gil, and Vinod Khosla—part of over 250 leaders driving 200+ sessions designed to accelerate your growth and sharpen your competitive edge. Additionally, connect with hundreds of startups pioneering innovation across every sector.

“ClothOff is explicitly designed and marketed as a deepfake pornography generator for images and videos,” Langford explained to me. “When you’re litigating against a general-purpose system that users can query for any purpose, the legal landscape becomes far more complex.”

Several U.S. laws, most notably the Take It Down Act, already prohibit deepfake pornography. Yet while specific users clearly violate these statutes, holding an entire platform accountable is considerably harder. Existing laws typically require proof of intent to cause harm, meaning demonstrating that xAI knew its tool would be used to create non-consensual pornography. Absent such evidence, xAI’s fundamental First Amendment rights could provide substantial legal protection.

“From a First Amendment standpoint, it is unequivocal that Child Sexual Abuse Material is not protected speech,” Langford notes. “Therefore, if you design a system specifically to produce that content, you are clearly operating outside First Amendment protections. However, with a general-purpose system that responds to diverse user queries, the legal boundaries are much less clear.”

The most straightforward legal path would be to demonstrate that xAI willfully ignored the misuse. This is a plausible argument, given recent reports that Musk directed employees to relax Grok’s safety guardrails. Nonetheless, even with that approach, pursuing such a case would carry far greater legal risk.

“Reasonable observers might argue that this risk was foreseeable years ago,” Langford says. “How could more stringent controls not have been implemented to prevent this? That suggests a level of recklessness or knowledge, but it still makes for a more complicated case to litigate.”

These First Amendment complexities explain why the most significant regulatory pushback against xAI has originated in jurisdictions without robust free speech protections. Indonesia and Malaysia have already moved to block access to the Grok chatbot, while UK regulators have opened an investigation that could lead to a similar ban. Preliminary actions have also been taken by the European Commission, France, Ireland, India, and Brazil. In contrast, no U.S. regulatory body has issued an official response.

It remains uncertain how these investigations will conclude, but the surge in AI-generated abusive imagery has undoubtedly raised critical questions for regulators—and the findings could be incriminating.

“If you are posting, distributing, or disseminating Child Sexual Abuse Material, you are violating criminal law and can be held accountable,” Langford states. “The more difficult legal question is: What did the platform know? What actions did it take, or fail to take? And what is it doing now in response?”