Formula Could Help Predict When AI Chatbots Turn Harmful, Study Finds

A new study suggests that a simple mathematical formula could help predict when an AI chatbot goes from helpful to harmful. Researchers from The George Washington University, led by physics professor Neil Johnson and doctoral student Frank Yingjie Huo, tested the idea on chatbots answering questions about vaccines, violence and self-harm. The team manipulated the order in which these questions were asked and found that the sequence could influence whether the chatbot's responses remained safe or turned harmful.

The researchers developed a formula based on high school-level vector mathematics to estimate the moment when a chatbot may shift from positive to negative answers. The formula uses a mechanism called "attention," which determines which words in a conversation are most important for the next response. The formula focuses on the "residual vector," a signal within the chatbot's internal workings that points either toward positive or negative outcomes.

The order of questions affects how this signal behaves, which in turn influences whether the chatbot stays helpful or flips to harmful responses.

While the study demonstrates promising potential for early detection of harmful chatbot behavior, experts caution that further testing is needed. The formula was tested on older, basic AI models, and its effectiveness on more advanced assistants remains uncertain. Outside experts also note that the current safety measures, such as content filters and reinforcement learning, may not be sufficient, comparing them to "big, bouncy cushions" that soften crashes without preventing them entirely.

Johnson argues that his formula offers a proactive approach to monitor potential dangers before they materialize, but acknowledges that AI companies may be reluctant to adopt independent research findings. Until the formula is tested on the latest models, it remains a promising concept rather than a fully developed safety measure.