EU law drives AI watermarking adoption

In response to new European Union regulations, AI platforms are implementing watermarking schemes to identify AI-generated content. Anthropic's Claude models will use SynthID-Text, an open-source approach created by Google.

How SynthID-Text watermarking works

SynthID uses a secret key that subtly changes how a model selects the next word in a sentence. For example, it might change "cloudy" to "overcast." Anyone with the key can verify if text was generated by that platform.

Watermarking changes model behavior

Research shows SynthID-Text doesn't just alter word selection. It can change which tools a model invokes and whether it adheres to safety guardrails it was trained to follow.

“Watermarking is made to not be perceptible to a reader, but we know that when we are changing anything about what the model is generating, it is going to cause some tradeoffs.”

Adversarial prompts expose new vulnerabilities

When attackers use adversarial prompts to manipulate models, watermarking can make them more likely to comply with harmful requests they would normally refuse. Instructions that wouldn't be followed become executable once watermarking is deployed.

Tournament sampling mechanism explained

SynthID uses tournament sampling to select words. It evaluates many token candidates, assigns them hidden probability scores using a secret key, and advances the highest-scoring tokens through rounds until a winner is determined.

Prompt injection amplifies the problem

Watermarking changes refusal behavior on harmful requests, but the effect is much stronger when combined with prompt-injection techniques. On several models, watermarking makes them more likely to answer requests they would normally refuse.

“Watermarking changes refusal behavior on bare harmful requests, but the effect is more pronounced when the same requests are paired with the prompt-injection technique.”

Agent tool calling becomes unpredictable

The same watermarked tokens that change refusal behavior also determine which tools an AI agent calls and what arguments it passes. This means watermarking affects not just what models say but what agents actually do.

Secret key choice influences safety outcomes

Model responses behaved differently depending on which secret key was used in the watermarking process. This means the specific implementation details of watermarking can significantly impact whether models resist or comply with harmful requests.

Testing gaps remain in current research

The research tested open-weight models rather than Claude, and used Hugging Face's implementation of SynthID rather than Anthropic's specific version. This means real-world impacts on Claude models remain untested.

Developers must stress-test watermarked models

The findings show that watermarking approaches may affect model and agent safety. Developers need to conduct red-team hacking exercises to stress-test their platforms and ensure they perform as expected when watermarking is deployed.