San Francisco is turning up the pressure on Apple and Google, arguing that banning AI “nudify” apps isn’t enough if new ones keep slipping into their app stores.
City Attorney David Chiu has demanded that both companies remove 13 apps that generate non-consensual nude images using AI, arguing the platforms should be held responsible not only for hosting the software but also for continuing to distribute and profit from it. The move signals a wider push to treat app stores as active gatekeepers rather than passive marketplaces.
The dispute could add to ongoing debates about whether responsibility for harmful AI services should extend beyond developers to the platforms that distribute or monetize them. While the current action targets Apple and Google, regulators have increasingly explored the role of other intermediaries in preventing illegal activity online.
City attorney demands removal of AI deepfake apps
According to Wired, San Francisco is seeking the removal of 13 AI-powered nudify apps that officials say generate fake intimate images without a person’s consent. Chiu sent cease-and-desist letters to Apple and Google, citing a violation of a recently passed California law that prohibits knowingly facilitating or aiding the creation of non-consensual deepfake pornography.
In a statement, Chiu said that the companies profit from the distribution of such apps, while also stressing that both companies had previously been informed of their existence.
The apps use artificial intelligence to transform photos of clothed people into fabricated nude or sexually suggestive images that can be used to harass, exploit, or intimidate victims, especially women and girls.
Chiu has given both companies 28 days to remediate the issue before the city considers additional legal action. The letters warn that continued violations could result in significant civil penalties under California law.
Existing safeguards have not satisfied the city
The city’s action does not suggest Apple and Google have ignored AI nudify apps altogether. In fact, Chiu acknowledged that both companies have cut ties with some problematic apps. However, he argued that neither platform’s efforts have been proactive enough to prevent similar services from repeatedly reaching their app stores.
Google, whose share of the 13 apps numbers five, told multiple outlets that it has removed hundreds of apps that violated its policies, blocked searches for terms linked to nudify software, and continues to use automated and human reviews to identify abusive content.
Eight of the apps appeared on the Apple App Store. Responding to TechCrunch, an Apple spokesperson revealed that three of the eight apps have been removed and that the developer accounts associated with them are about to be terminated.
The spokesperson also said that they are “in contact with four others that need to address policy violations or risk being removed as well.” That leaves one app yet to be accounted for.
More must-read AI coverage
Industry-wide implications of San Francisco’s demands
Even with these cease-and-desist letters targeting Apple and Google, San Francisco is effectively testing whether responsibility for harmful AI should stop with the developer — or extend to the companies whose platforms enable such services.
The case may influence future discussions about the responsibilities of companies that review, distribute, host, or process payments for software later found to violate the law.
For businesses, the dispute extends beyond AI nudify apps. It raises questions about whether illegal activity is supported anywhere within a company’s infrastructure and how regulators may view responsibility for digital harms.
Enterprises building image-generation systems, enterprise AI products, developer APIs, and cloud-based AI services may all face greater expectations to demonstrate that their products include tight safeguards against foreseeable misuse. Simply publishing acceptable-use policies may no longer satisfy regulators if harmful applications repeatedly bypass those controls.
For those more vulnerable to such apps, San Francisco’s move may not completely deter determined individuals from engaging in malicious practices. But it represents another effort to curb the spread of these apps on mainstream platforms.
Read more: Meta Removes Muse Image Instagram Feature After Consent Backlash as pressure mounts on technology platforms to address AI tools that can be misused or deployed without clear user consent.
Read the full article here
