According to a report by China National Radio broadcasted on September 8, a former game company programmer named Zheng has been convicted by the Yuyao City People’s Court in Zhejiang Province for creating and selling obscene items through generative artificial intelligence. Zheng and a six-person team bypassed model safety filters using prompt engineering, role cards, and dialogue scenarios, generating over 370,000 static images and 6,000 dynamic images while collecting 24,000 dollars in user payments.
How the Generative AI Model Was Built and Monetized
The operation began in July 2024 when Zheng encountered an overseas AI chat robot and decided to develop a similar software for financial gain, according to court details. Following debugging, the team launched the generative AI model on an overseas platform in January 2025.
To collect revenue, the creators integrated fourth-party payment systems and virtual currencies. Users who purchased recharge packages could select character cards and interact with AI robots in sexually explicit chats. These conversations allowed users to spend accumulated points to generate explicit static and dynamic imagery.
Investigation, Evidence, and Legal Findings
Public security officers initiated an investigation after discovering illegal clues in July 2025, arresting all six defendants over a two-month period. Criminal trial division court assistant Sun Yuiping stated that the operation produced more than 370,000 static images and over 6,000 dynamic images. Forensic identification of a randomly sampled 12,500 static pictures and the entire pool of dynamic files revealed that over 9,000 static images and 5,000 dynamic images constituted obscene materials, while user recharges totaled approximately 170,000 RMB based on the 24,000 dollar figure.
Did You Know? The generative AI model operated for several months before law enforcement launched a formal investigation in July 2025, ultimately arresting all six members of the development team within a two-month span.
Criminal trial division judge Zhu Zhanglin noted that standard generative AI models possess built-in safety filters designed to block illegal content, meaning the base software is not inherently a tool for crime. However, the defendants actively used prompt engineering, edited system instructions, and set up role-play settings to bypass these security protocols. Zhu Zhanglin emphasized that technical neutrality does not shield developers from criminal liability when they deliberately dismantle model safeguards and retain control over the output environment.
By actively engineering prompts to dismantle built-in safety filters, creators forfeit claims of technical neutrality and take direct legal responsibility as the producers of any illicit material generated through their systems.
Regulatory Context and Future Legal Enforcement
The Yuyao City People’s Court ruling aligns with legal standards concerning artificial intelligence disputes. On September 7, the Supreme People’s Court issued an opinion requiring strict punishment and criminal liability for individuals who use deepfake and related technologies to commit crimes, including the production, sale, and dissemination of obscene articles.
Frequently Asked Questions
Who initiated the generative AI software project?
How much revenue did the illegal AI platform generate?
Users paid a total of 24,000 dollars, which translates to approximately 170,000 RMB, through fourth-party payment channels and virtual currencies integrated into the overseas platform.
What legal charges did the defendants face?
The Yuyao City People’s Court convicted Zheng and his co-defendants of the crime of producing obscene items for profit, holding them criminally responsible as producers despite the content being generated in response to user prompts.
How should technology platforms balance open-ended prompt customization with the prevention of illegal content generation by users?
Keep reading