Understanding the Impact of LLM Watermarking on AI Agent Behavior

Recent announcements reveal that future Claude models from Anthropic will incorporate an imperceptible watermark in their output, based on Google DeepMind's SynthID-Text [2, 3]. While text watermarking is not entirely novel, its practical implications carry regulatory significance. The EU AI Act [4] mandates that AI systems generating synthetic text must mark their outputs in a machine-readable format detectable as artificially generated or manipulated, using reliable and interoperable techniques.

The SynthID-Text watermark alters the model's token generation process, potentially impacting safety behavior, such as whether the model declines to fulfill harmful requests and whether such rejections persist under prompt injection. At the agent level, the same tokens can dictate which tools are invoked and the arguments passed to them.

Consequently, watermarking can influence both the content of the model's responses and the actions of agents utilizing these models. This phenomenon, termed sampling drift, is an empirical question. Our findings indicate that both model refusal behavior and agent tool calling are affected, albeit the impact varies by model and key.

We report both aggregate performance and pairwise disagreements between watermarked and unwatermarked runs. Employing SynthID's non-distortionary configuration, we observed no significant quality degradation across nearly twenty million Gemini responses [3]. Anthropic's watermarking strategy applies at the model level, affecting supported models accessed through the Claude Platform API and cloud providers [1].

Developers employing watermarked models as reasoning components in agents may receive watermarked outputs even if the agents themselves remain separate applications, making model-level behavioral effects pertinent to agent-level design. Tournament sampling, with its greater capacity to alter token selection during uncertainty, can also modify arguments executed by agents when the model's weights and prompt stay constant.

Non-distortionary watermarking does not guarantee identical behavior under a fixed key, as individual generations can diverge despite identical watermark randomness. Our paired experimental design evaluates tool calling on BFCL v4 single-turn AST [9] and refusal on 200 HarmBench harmful behaviors [10] and 100 benign JailbreakBench controls [11].

Using SynthID-Text through HuggingFace's unmodified SynthIDTextWatermarkLogitsProcessor, we generate items with and without watermarking from the same seed, batch composition, and order at various temperatures. Individual call evaluations reveal that watermarking reduces accuracy on six of the seven models, with notable decreases on four.

However, aggregate results may appear unchanged due to opposing changes in accuracy and incorrect calls potentially being offset by correct ones. Pairwise disagreements, or churn, quantifying the share of items with differing verdicts between watermarked and unwatermarked runs, are measured to assess the extent of behavioral drift induced by watermarking.