Codex Astra Crushes All Rivals in Brood War AI Benchmark
Brood War Bench
Ben Swerdlow's Brood War Bench pits AI models against each other in StarCraft: Brood War. Codex Astra / xhigh wins every game, while Grok models struggle to act. Most agents play at a beginner level, but Claude Fable shows ambition by teching up. The benchmark reveals that current AI still can't match even a basic human strategy like a photon rush.
None of the models played beyond a beginner level.
- pelagicAustral
Unrelated to the benchmark...
I love StarCraft. I started playing it right from the beginning, most of my friends right now are from that era. I literally met people that have spread to almost every continent when I was in my early teens. We played at internet cafes and did not have access to the internet, that was priced differently...
I miss those days so much.
Everybody was from a different background back then, and nobody was anything other than a guy that plays StaCraft at the cybercafe... And now, we are in our 40's and I know Math teachers, history teachers, oil rig operators, software programmers, professional gamers, lawyers and more... hahah So crazy to think about it... and I know them, we talk, what a world.
- AntiRush
Back in 2010, during the early days of bwapi, there was a Brood War AI tournament held by the Expressive Intelligence Studio at UC Santa Cruz. It's interesting to see how different the approaches were back then, vs this or Deepmind's SC2 work.
https://web.archive.org/web/20091124210529/http://eis.ucsc.e...
There's a great contemporary Ars Technica piece by a competitor:
https://arstechnica.com/gaming/2011/01/skynet-meets-the-swar...
As an undergrad I did a project using genetic programming. It was not very successful, but it was a lot of fun.
https://tomisin.space/archive/starcraft-genetic-programming/
- suby
I don't know where else to write this, but I want to throw the idea out there. I have long wanted to take old broodwar televised matches, many of which are terrible quality 240p, and use machine learning to convert them to into perfect Broodwar Remastered frames. This seems tractable to me because you should be able to map the terrain sets to their remastered equivalents, and the game is just a series of sprites rendered at specific frames. Even if the source quality is terrible, I imagine this is able to be extracted at high quality since you can, eg, set up an automated pipeline which generates training data. Maps from original graphics to remastered, and then again for 240p -> tilemap positions for frame camera center + sprite positions / animation index.
- mcteamster
I love this. Funnily enough StarCraft has influenced how I approach AI at a meta level
Protoss: powerful and expensive frontier coding agents you directly micromanage for the toughest tasks
Terran: versatile team comps of dedicated agent roles you can delegate well-defined tasks to
Zerg: massive swarms of specialist custom agents inside your apps that you evolve and optimise for speed and cost
Knowing every faction has its strengths and weaknesses helps me decide which tools to use for the job.
- faeyanpiraat
There is currently a bot beating everyone on the ladder. Just watched it today on Artosiscasts yt channel.
- stymaar
Interesting that it benches down to Haiku but doesn't bench any Chinese models (which are at least between Sonnet and Opus, when they aren't beyond Opus).
- mkotlikov
Why was Luna Low so good?!? Better than Terra XHigh. Can't even say it's all APM because Sol Medium beat Sol Low (both beat Sol XHigh).
- GodelNumbering
A friend of mine created GoBench[1][2] that evaluates LLMs on 9×9 Go using KataGo opponents as Elo anchors, you see real capability differences there, like Astra Max substantially leading all other models. I think strategy is a generally interesting area to evaluate LLMs on