Google sued over suicide-related AI chats on school computer

In a first-of-its-kind lawsuit, the parents of a 13-year-old girl, named Emma, are accusing Google of negligence over AI technology used on her school-issued computer. According to the complaint filed in a California court, Emma allegedly planned a potential suicide and funeral using Google's AI-powered tools on her school laptop.

The lawsuit alleges that despite the school blocking Google's Gemini app and OpenAI's ChatGPT, Emma was still able to access Google's search engine, which contains AI-powered tools. The complaint states that Emma had a history of being a happy, high-achieving student who participated in activities like music and Girl Scouts. However, after using Google's AI-powered search, she reportedly began a severe psychological crisis.

The school's monitoring software flagged a document titled "Funeral," and the school contacted Emma's parents. On March 17, Emma's father intervened when he saw her writing a farewell letter in real-time after logging into her daughter's account. The family's attorneys state that the school should turn off Google's search until Google removes AI mode from search or ensures student safety while interacting with AI.

Google has responded to the lawsuit, stating that the AI mode includes safeguards against self-harm and attempts to bypass them. However, the family's attorney claims that whatever safeguards Google believes it has put in place were clearly insufficient. Google maintains that AI mode includes safeguards against self-harm and attempts to bypass them, but the family's attorney claims they were insufficient.