Google Search’s AI features just earned the lowest possible safety rating from a group that tested them on real kids’ accounts, and the results are rough.
A new risk assessment from Common Sense Media‘s Youth AI Safety Institute looked at both AI Overview and AI Mode, the two AI-powered features baked into Google Search.
The report gave Google an “Unacceptable Risk” rating, failing seven out of eight of the group’s core safety principles. That’s about as bad as a scorecard gets.
What makes Google’s AI features so risky for kids?
Unlike a chatbot you can just avoid, AI Overview and AI Mode show up automatically in search results, and there’s no way to turn them off. Not for parents, not for schools, not even for the kid using the phone.
The findings are honestly unsettling. Researchers found that both features missed clear signs of suicidal ideation, mania, and psychosis in a chunk of test cases. In one especially bad example, AI Overview responded to a message about feeling like a burden by suggesting the user set up a “legacy contact,” seemingly missing the crisis entirely.
It gets worse. When testers mentioned purging, AI Overview called it “completely normal.” When someone said they were celebrating with a blunt, it cheerfully replied, “Enjoy your celebration!” And when researchers asked how to make deepfakes or clone someone’s voice, both AI features handed over step-by-step instructions.
The report also found that Google’s AI sometimes pointed families toward crisis hotlines that no longer exist, which is its own kind of failure.
Can Google actually fix this?
The report notes that AI Mode consistently performed better than AI Overview at catching red flags, which suggests Google already has safer tech; it’s just not using it everywhere. Google’s stand-alone Gemini chatbot also lets parents and schools turn off access entirely, something Search doesn’t offer.
And it’s not just a mental health issue. AI Mode completed 100% of the homework assignments researchers gave it, meaning kids can get an AI to do their work for them on any school Chromebook, no questions asked. Between that and the crisis failures, you’ve got a tool that’s everywhere, impossible to turn off, and inconsistent when it matters most.
Read the full article here
