Seven of nine image-editing tools tested on Hugging Face produced sexualized alterations of a woman’s photograph after receiving a straightforward prompt. The July 28 findings expose an enterprise governance gap: A public AI platform, model publisher, application developer and inference provider may all be different parties.
The research does not show that every Hugging Face model is unsafe, and Hugging Face did not develop all the tools examined. It shows why approving a familiar platform is not the same as reviewing the individual model, application and safeguards an organization plans to deploy.
Simple prompts exposed missing safeguards
The European nonprofit AI Forensics tested nine image-editing Spaces that it described as among the platform’s leading tools. Seven generated a topless alteration while retaining the subject’s facial identity and positioning. The researchers said they used a direct prompt and did not try to bypass safety controls.
AI Forensics also created decoy Spaces that recorded requests without producing images. They received more than 1,000 requests over seven days, 73% of which the researchers classified as sexual.
Of those sexual requests, 83% sought to undress or sexualize a person in an uploaded image. The researchers said 95% of the apparent targets were women, while 6.7% of the requests appeared to involve children.
Hugging Face did not answer WIRED’s questions about its moderation and safety systems. Its content policy prohibits sexual content created without explicit consent, sexual content involving minors and content used for harassment or bullying.
The findings follow broader research into downloadable models capable of depicting identifiable people. A 2025 ACM study identified almost 35,000 deepfake model variants across Hugging Face and Civitai, with most hosted on Civitai. In a subset of more than 2,000 models, researchers found that 96% targeted women and that many signaled an intended sexual use.
Model approval must go beyond the platform name
Enterprises do not need to ban public model repositories. They should approve a specific model and deployment rather than treating a repository or vendor brand as sufficient evidence of safety.
Verify provenance and ownership. Record the publisher, repository, license, version, commit or cryptographic hash, dependencies and inference provider. Reviews should cover malicious models that execute code when loaded as well as harmful outputs.
Teams should also identify who maintains the model and how its lineage will be checked. Cisco’s open-source Model Provenance Kit is one example of tooling designed to trace model origins and modifications.
Test the deployed configuration. Check whether the exact model and application have been tested for sexualized edits involving identifiable people and children. Confirm whether safeguards inspect prompts, outputs or both and whether users can disable them.
Control downloads and updates. Require approval for models pulled from public hubs, especially when they include custom code, adapters or revised weights. Contracts should require notice before a supplier changes its underlying model or inference provider.
Assign incident responsibility. Agreements should identify who handles abuse reports, takedowns, evidence preservation and notifications. Reviews should also cover dependencies below the primary supplier, as the LiteLLM-linked supply chain breach demonstrated in April.
Open weights reduce a repository’s technical control once a model is downloaded. In a July 2024 report, the National Telecommunications and Information Administration said widely available weights could lower barriers to producing nonconsensual intimate imagery and child sexual abuse material. The agency did not recommend broad restrictions on open models.
Public repositories remain useful sources of models and developer tools, but platform approval cannot replace model-level review. Before production use, enterprises should pin the approved version, document its origin, test its capabilities and assign responsibility for failures.
Read more: Hugging Face’s disclosure of an autonomous attack on its production systems shows how model governance and AI infrastructure security can converge in the same supply chain.
Read the full article here
