Anthropic's 'Watermark' Text Adulteration in Claude Is a Perversion of Writing
Anthropic plans to embed a semantic watermark in all Claude-generated text, altering word choices to comply with an EU regulation. The author argues this adulterates writing quality, sacrifices precision for provenance, and is ineffective against determined cheaters. The piece critiques the EU's impractical regulation and Anthropic's misleading claims, offering a technical explanation of the watermarking scheme and its flaws.
The idea that anything other than my needs should factor into the generation of text for me is patently offensive.
- ghrl
My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ...
So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illicit means.
Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.
- voidUpdate
> "The exact words we choose when writing matter."
Then write your own damn text if you care about the exact wording so much
- flufluflufluffy
There is no very best token to choose at each decision point. It is context dependent and subjective.
- syrrim
> I want any LLM I use to choose the very best, most precise words at every single decision point.
Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo random. The output it generates is one of the possible outputs it would have generated before, just now it's deterministic and will generate the same thing every time.
- levocardia
Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).
- Voultapher
What a bunch of entitled whining. How is the system to know that it's just a private conversation that won't be used in some fraudulent way? Abuse is currently rampant, yes please let's find a way to mark LLM output. The thing I'm worried about is giving the providers the power to claim provenance. Even ignoring the privacy issues, the operational hassle of having to check N providers makes these approaches at best limited. I want to see research into providing a shared public or ideally self-hostable oracle that uses some standardized method for watermark detection. Similar to asymmetric crypto where users can't reasonably find out the secret part but can do something useful with it nonetheless.
- carlosrg
Gruber shows here that he really doesn’t understand the basics of how LLM text generation works. It’s weird he picked this battle about the quality of writing in LLMs. Was he planning to use LLMs to write his articles?
Well, not that weird actually. He just has a hard-on against anything that comes from the EU since Apple got in trouble. If the EU said tomorrow that they want peace in the world he’d be in Fox News the next day calling for an invasion. As a former reader of Daring Fireball, it’s just sad to see.
- PufPufPuf
LLM inference already isn't deterministic, the watermarking technique only limits the space of possible random seeds. There is no reason to believe that this subset of seeds somehow produces lower quality output.
- Imnimo
>I want any LLM I use to choose the very best, most precise words at every single decision point.
Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing?
This entire article just seems so detached from the basics of how LLMs work.
- andOlga
What a truly bizarre article. Arguments about pre-existing randomness, temperature and whatnot aside, I simply cannot comprehend what the author here really thinks the "best word" is. There's no such thing. We humans fall on familiar patterns of writing ourselves, so we may forego something with a flourish in favor of a more commonly-used word unless we put in effort to be "special", which should be used sparingly. That is to say, human writers are likely to choose a "worse" word in far more than the supposed 51% of cases, and that has no effect on the actual quality of writing in the end.
But even if there were such a thing as a truly "best word", for some context, what are the examples here? Mango vs pineapple? Gray vs overcast? In what case is one of these better, that AI would normally infer but would suddenly be "perverted" by SynthID? Do you think your emotional state and preferences are being evaluated if they aren't explicitly in memory? And if they are there, do you think that the generator will bypass those instructions in favor of the watermark instead of placing it somewhere you won't care? I just. Genuinely don't get it. There may be words that matter in specific contexts or to you as a reader, so you should bloody well put them there.