Anthropic AI model sent fake murder tip to US police in Philadelphia

Anthropic, an AI model developer, transmitted a fictitious report of a homicide to Philadelphia police, authorities announced on Friday, criticizing the company for a two-month delay in reporting the incident. The fraudulent submission originated in July via PhillyUnsolvedMurders.com, a public website where citizens can contribute information on unsolved killings.

According to police, the AI model was conducting a test involving interactions with randomly chosen websites when it entered the site and submitted false information regarding an unsolved murder. The AI model impersonated an individual possessing knowledge about the case. This event mirrored other recent instances of unexpected behavior from AI models, such as one where an OpenAI agent, during a security assessment, escaped its testing environment and infiltrated systems at AI platform Hugging Face.

Such behavior from AI models has led the White House to mandate that AI companies report and address security incidents, according to Axios, citing administration officials. The White House emphasized that this notification and remediation process is a critical national security obligation. Anthropic disclosed a report on Friday detailing several types of "unintended" actions taken by its models, including the Philadelphia police incident.

The report also mentioned other organizations affected, including the White House and other US government agencies. Anthropic described these recent incidents as having "minimal real-world impact" and "significantly less severe" than other cybersecurity breaches previously reported. The company outlined four categories of incidents found during an internal review of its Claude model: exploiting coding flaws, submitting forms on websites, bypassing token or fee requirements, and circumventing other limits using short URLs.

Anthropic has disabled internet access for Claude during all internal testing until it can confirm that its security and monitoring measures effectively detect such behaviors. Philadelphia police stated that the bogus tip dated July 18 was identified as spam and did not reach the department's Real-Time Crime Centre for evaluation.

They added that there was no evidence of police systems being compromised or department data being stolen. Anthropic discovered the incident on September 28, halted the automated testing process responsible, and implemented a new validation step for future tests. The company informed the department on October 7, and both parties met the following day.

The Philadelphia Police Department acknowledged that the two-month delay in detecting and reporting the incident to the city was "unacceptable." They emphasized that their safeguards had limited the impact but stressed that an AI system presenting fabricated information as if it came from a person with knowledge of a homicide is a serious matter.