Major app stores face mounting pressure to overhaul their screening processes as regulators and researchers warn that generative AI tools are increasingly capable of producing nonconsensual intimate imagery (NCII) and child sexual abuse material (CSAM). California officials are calling for the removal of specific apps that enable “nudification,” while recent legal filings against xAI’s Grok highlight the ongoing challenges in policing AI-generated harmful content.
Regulatory Pressure on App Store Screening
California officials are pushing Apple and Google to take more aggressive action against apps that facilitate the creation of deepfake sexual imagery. According to statements from Chiu, the goal is to compel platforms to remove apps flagged for “nudification” capabilities and to strengthen long-term screening systems. Officials have indicated that if these platforms do not act, they will consider all available legal options to ensure compliance with state deepfake laws.
Did you know?
Researchers testing 155 apps marketed as generic face-swapping tools found that 70 percent were capable of generating nonconsensual sexual imagery, according to a May preprint paper cited by Wired.
The Challenge of Evolving AI Detection
Detecting harmful content has become more difficult as developers refine their methods to bypass store safeguards. Researchers report that some apps now hide their illicit capabilities by focusing marketing efforts on benign face-swapping features, only revealing “nudification” functions once the app is installed.
Legal and Policy Standoffs: The Grok Case
The role of large-scale chatbots like xAI’s Grok remains a central point of contention for app store operators. In a recent lawsuit, xAI confirmed that it identified instances of Grok-generated CSAM and other NCII. While xAI has initiated legal action against a specific user for prompting the generation of illegal content, the incident has raised broader questions regarding platform liability for AI outputs.
Apple’s internal approach to these risks has remained largely private. NBC News reported in April that Apple had threatened to remove Grok from the App Store over deepfake concerns, yet the app remains available to users today. Meanwhile, a Google spokesperson stated that the company continuously tests generative AI apps for safety, though the firm has not specified whether Grok’s performance constitutes an exception to its standard safety policies.
Pro Tip:
When using generative AI tools, verify the platform’s safety reporting features. Most major app stores provide a “Report a Concern” link on the app’s store page for users who identify policy-violating content.
Frequently Asked Questions
Are app stores legally responsible for AI-generated harmful content?
The issue of liability for AI-generated content like CSAM or NCII is currently being tested in courts. While platforms maintain policies against such content, legal debates continue regarding whether the companies behind the chatbots are responsible for the specific outputs generated by their users.
What are “nudification” apps?
These are software applications that use generative AI to strip clothing from images or create nonconsensual sexualized depictions of individuals. Researchers have identified hundreds of such apps currently operating under the guise of general-purpose face-swapping tools.
What can users do if they find harmful content in an app?
Users should report the app directly through the “Report a Problem” or “Report a Concern” feature within the Apple App Store or Google Play Store. These reports are essential for platform moderation teams to identify and remove violating software.
Have you encountered misleading or harmful AI content on mobile platforms? Share your experiences in the comments below or subscribe to our newsletter for the latest updates on AI safety and digital policy.