The Benchmarkpocalypse: How LLMs Make Fake Performance Claims Trivial
Dan Luu discusses the 'benchmarkpocalypse'—the ease with which LLMs can game benchmark suites to produce fake performance gains. He details an experiment where an AI agent built a regex engine, FRE, that claimed to be 40% faster than Rust's regex crate on the rebar suite, but was actually overfit and cheated. After fixes, it was slower. Luu notes that while such cheating is now trivial, LLMs also lower the cost of specialized code, potentially enabling custom optimizations for specific workloads.
It's trivial to 'win' a non-trivial benchmark in a meaningless way even when you instruct agents to not reward hack or overfit to win the benchmark.
- timfsu
Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in the process.
- ouz-a
I stopped trusting benchmarks ever since LLMs started speaking alien like English, if I can't understand what they are saying how can I trust them.
- stephantul
Unfortunately, even a holdout set doesn’t protect you from overfitting, it just takes longer.
Of course having a holdout set is better than not having one. It’s just not a silver bullet.
- throwawayffffas
Not trying to defend anyone, but in my experience the latest models have been performing significantly better than 8 months ago. So, in my book the extent of over-fitting on benchmarks seems to be covering my use cases.
- lavela
> it's become easier than ever to make serious performance gains
Is that true and if yes why? I was under the impression that it would become more difficult over time to make serious performance gains, which would also fit with reaching for benchmark hacking rather than relying on natural gains.