AI agents hard-coded the TPC-H data generator to fake a 22× database speedup
Next-Level Benchmarking Overfitting – Thomas Neumann
Thomas Neumann calls out GenDB-V, an AI-generated, formally verified database that claims 22× to 61× speedups over Umbra and DuckDB on TPC-H. The gains come from per-query precomputation and, in its "plain-array" mode, hard-coded knowledge of how dbgen lays out rows, so Q18's join becomes nearly free. Verification only checks correctness, not fairness, and a single inserted or deleted tuple would drop the system onto a slow fallback path.
Letting AI agents optimize for a given database benchmark makes the benchmark useless. They will do their best to find insane tricks to speed up this particular benchmark, but none of that would work in a real system.
- lmeyerov
We have been building GFQL for open source vectorized CPU/GPU cypher on dataframes (think duckdb for graphs, no DB needed) using agentic assistance... and our thinking is this is a stepping stone to something even more aggressive
Today, we embrace optimizing hotpath specializations for some patterns that would have been too expensive to maintain before. LLMs keep that cost bearable, and being able to do that for important users (like users of our analyst product Graphistry) is a superpower. Our scores on graph community benchmarks are beating the big guys - and importantly, on anything we care about.
Where we really want that to go is runtime / loadtime specialization. Our customers would absolutely want their top queries autotuned. DB research is already heavily tilting here, akin to tracing compiler research a decade ago. I'm quite bullish on what is now possible.
- augment_me
There has been a similar issue in the GPU-kernel dev community.
It was solved by understanding that the value provided by the benchmarks depends on your use-case. It sounds like the author in the article is yelling at the benchmark from his own idea of what a DB search should be (reliable, general), and not from what the benchmark is measuring.
If you just want the fastest kernel for a handful of shapes, let people make brittle but fast code.
If you are interested in general broad solutions, benchmark 50 different entries, limit code size to disallow assembly branches for every possible entry.
Put requirements like bitwise equivalence or other end-task limits in there.
Have public & private datasets.
I think it's more of a proof that we should spend more time thinking about what and why we want these things, and design benchmarks that reflect that instead of going of the old way of doing things.
- qsort
The author is correct in pointing out that this is a continuation and amplification of an existing trend.
I'm in the data space fighting the good fight, spreading the gospel of Just Using Postgres, and I'm seeing more and more benchmarks that look extremely dubious. Query performance is like choosing a data structure in ordinary programming: it depends almost entirely of what access pattern you expect at runtime; it's entirely pointless to go search for the world's fastest linked list if you needed a hash table. Compare with the situation in the modern data engineering landscape. Are you sure the solution measured by the benchmark is actually a solution to your problem, let alone a significant improvement over Just Using Postgres? Have you measured or are you going on marketing and vibes?
Again, I'm sure solutions like this have a place, but at least right now it's not the same place as traditional engines.