Dan Luu Exposes Why Popular Benchmarks Like Senior SWE-Bench and Napkin Math Are Misleading
Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires
Dan Luu dissects three benchmarks: napkin math performance estimates, AI model evals like Senior SWE-Bench, and winter tire claims. He finds flawed code in napkin math—random memory reads measured without data dependencies, unaligned SSD reads—and questions whether leaderboards like Senior SWE-Bench truly measure general capability. The winter tire example shows how conventional wisdom can be misleading.
All-season tires lose traction and stiffen in freezing winter temperatures. Their rubber compounds are designed for warmer weather and become hard below 7°C (45°F), leading to significantly longer braking distances and reduced grip.
- cb321
Something Dan does not observe in his article (perhaps Jamie does elsewhere? edit: or even Dan elsewhere) is that the same problem which makes the memory latency benchmark unrealistic (or at least misleading) often impacts hash table lookup benchmarks as mentioned at https://github.com/c-blake/bu/blob/main/doc/memlat.md and probably many other benchmarks. Essentially, CPU work prediction/speculative execution has become so good that much care is often required to measure latency rather than reciprocal throughput. This all started in the 1990s (or probably earlier with Cray), but I guess there's been an ongoing educational failure/oversimplification tendency.
Of course, throughput at one "level of work" is often latency at a higher level (like command-to-command execution, for example). So, "what matters" might be throughput or "latency". It all depends. :-) People often fiddle with such semantics to market methods, products, ideas, ... and marketing is often at cross purposes with understanding.
- jbellis
While I'm generally sympathetic to the idea that the public coding benchmarks are inadequate, to the point that I've written my own in the past and will likely do so again, the complaint here is that "these tasks don't match what I do in my day job" which is ~always going to be the case. The hope with benchmarks is that you can capture properties that generalize, from examining performance against small set of tasks, and I do think that this is at least directionally true for well-designed evals.
(There's https://withspecific.com/benchmarks/real-swe but since the tasks are private we still don't really know what they're representative of.)
- dom96
This is great. I've been building my own model benchmark lately and it has indeed been so easy to mess up the scoring. It's simply much harder to come up with an algorithm that combines all your individual scores into something that isn't broken in some special circumstances. That's why I think many just start capping the results.
- stephantul
The (relevant) segue into tires elevates this post so much. I don’t understand why, but it does
- jdw64
I feel like AI development is hitting a wall now. Evaluations are driven by benchmarks, but I have no idea who is even evaluating the validity of these benchmarks. The idea that continuously training and scaling models will automatically yield broad capabilities across multiple domains hardly seems true anymore.
Benchmarks like Senior SWE-Bench apply discontinuous and arbitrary thresholds to evaluate results. If the generated code is semantically identical to the reference solution but even slightly exceeds a length cutoff, it fails? That seems like a genuinely flawed criterion, and the LLM-based graders themselves feel highly unstable.
Honestly, my impression is that LLM advancement is now completely dictated by benchmarks. But looking at recent trends, token prices are skyrocketing while coding capabilities haven't shown any massive improvements beyond a certain model generation. In fact, comparing GPT-6 Astra to 5.6 SOL, SOL often writes better code.
Considering all this, while training across diverse domains can pack a model with various pieces of knowledge, ultimately, it feels impossible for this approach to do something like derive the theory of relativity from medieval knowledge.