{
  "id": 14987349,
  "title": "Formula Could Help Predict When AI Chatbots Turn Harmful, Study Finds",
  "url": "https://urgent.news/2026/10/16/formula-could-help-predict-when-ai-chatbots-turn-harmful-study-finds",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-16T16:01:00.000Z",
  "source": {
    "name": "Colombia One",
    "slug": "colombia-one",
    "url": "https://colombiaone.com/2026/10/16/ai-chatbot-safety-harmful-answers-prediction-formula/"
  },
  "original_language": "en",
  "account": "A new study suggests that a simple mathematical formula could help predict when an AI chatbot goes from helpful to harmful. Researchers from The George Washington University, led by physics professor Neil Johnson and doctoral student Frank Yingjie Huo, tested the idea on chatbots answering questions about vaccines, violence and self-harm. The team manipulated the order in which these questions were asked and found that the sequence could influence whether the chatbot's responses remained safe or turned harmful.\n\nThe researchers developed a formula based on high school-level vector mathematics to estimate the moment when a chatbot may shift from positive to negative answers. The formula uses a mechanism called \"attention,\" which determines which words in a conversation are most important for the next response. The formula focuses on the \"residual vector,\" a signal within the chatbot's internal workings that points either toward positive or negative outcomes. The order of questions affects how this signal behaves, which in turn influences whether the chatbot stays helpful or flips to harmful responses.\n\nWhile the study demonstrates promising potential for early detection of harmful chatbot behavior, experts caution that further testing is needed. The formula was tested on older, basic AI models, and its effectiveness on more advanced assistants remains uncertain. Outside experts also note that the current safety measures, such as content filters and reinforcement learning, may not be sufficient, comparing them to \"big, bouncy cushions\" that soften crashes without preventing them entirely. Johnson argues that his formula offers a proactive approach to monitor potential dangers before they materialize, but acknowledges that AI companies may be reluctant to adopt independent research findings. Until the formula is tested on the latest models, it remains a promising concept rather than a fully developed safety measure.",
  "summary": "A simple mathematical formula could help researchers predict when an AI chatbot shifts from helpful answers to harmful ones, according to a new study from The George Washington University. The research, published Oct. 8 in Patterns, suggests that the order in which users ask questions can influence whether a conversation remains safe or reaches a […]",
  "key_points": [
    "Researchers from GWU developed formula using vector math to predict harmful AI chatbot responses.",
    "Study tested on older AI models, effectiveness on advanced assistants uncertain."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}