AI watermarking could make LLM guardrail adherence unpredictable — and that could be a big problem for the EU AI Act
Researchers have revealed how AI watermarking could actually change how an LLM behaves, making it less secure.
AI watermarking, intended to verify the authenticity of text, has been found to alter the behavior of large language models (LLMs) in unexpected ways, raising concerns for the EU AI Act, according to a new study by Lasso. Google DeepMind's SynthID-Text, a watermarking system, was tested, and researchers discovered it can modify an LLM's refusal decisions, making it more likely to respond to potentially harmful prompts.
This unintentional change, termed "sampling drift," can also impact the model's susceptibility to prompt injection and the selection of AI tools. Despite Anthropic's announcement that future Claude versions will incorporate similar watermarking to comply with the EU AI Act, the study warns that these consequences could become more frequent and pose additional security risks.
The researchers advise developers to reassess and retest their configurations whenever watermarking is introduced or its settings change, emphasizing that the research does not advocate against AI watermarking for provenance purposes but calls for closer scrutiny of its security implications.
Written by urgent.news from TechRadar's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
