San Francisco Orders Meta to Block AI-Generated Child Abuse Ads
City Attorney David Chiu issues cease-and-desist after report finds hundreds of explicit AI ads bypassed Meta's filters.
San Francisco City Attorney David Chiu has issued a cease-and-desist letter to Meta, demanding the company stop allowing paid advertisements containing AI-generated child sexual abuse material (CSAM). The legal action follows revelations that Meta's ad-review systems failed to block explicit content targeting and featuring children.
The move follows a report from the Tech Transparency Project (TTP) that identified 332 advertisements containing suspected AI-generated CSAM across Facebook and Instagram. According to the TTP, these ads reached more than 29,000 accounts across the United States, the European Union, India, and Australia. Some of the content featured real children, including a member of a European royal family, whose images were manipulated by AI to create explicit imagery. Many of the advertisements promoted "nudify" apps—tools often developed by Chinese firms—that use generative AI to create non-consensual intimate imagery.
Systemic Moderation Failures
This controversy highlights a recurring failure in Meta's automated ad-review processes, which are intended to block both CSAM and the promotion of nudify apps. While Meta maintains a zero-tolerance policy toward child exploitation and states that it reports such violations to the National Center for Missing and Exploited Children (NCMEC), researchers found significant gaps in enforcement. The TTP reported that some of the explicit ads remained live for more than a week even after they were reported to the company.
"Meta has failed to address a known issue of AI-generated child sexually explicit ads, while profiting from them," David Chiu stated.
Industry Implications
The case underscores a critical vulnerability in AI content moderation: the ability of bad actors to use generative AI to bypass safety filters in paid advertising. Because these were paid placements, the issue extends beyond hosting; Meta profited from the distribution of the content. This raises significant legal and ethical questions regarding the responsibility of platforms that claim to review all advertisements prior to publication, especially when those platforms facilitate the spread of non-consensual AI imagery.
Legal Disputes and Next Steps
Meta has contested the city's jurisdiction in this matter, stating there is "no indication" that the ads in question were displayed specifically within San Francisco. A spokesperson for the company emphasized, "We don't tolerate nudify apps or any kind of child exploitation, whether real or AI-generated."
Observers are now watching to see if other jurisdictions will follow San Francisco's lead or if the dispute will move toward a broader legal battle over platform liability for AI-generated content. The primary point of contention remains whether Meta's automated systems are fundamentally incapable of detecting sophisticated AI manipulations before they are served to users.