FrontierHarness Eval – Compare AI coding harnesses fairly
Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
FrontierHarness Eval is a comprehensive benchmark that evaluates nine AI coding harnesses—including Codex, Claude Code, and Exo Harness—under identical conditions. Using the same model (Kimi K3) and runtime, it runs 360 trials from fresh checkpoints to eliminate warm-cache bias. The results reveal surprising cost disparities: Exo Harness costs just $1.05 per task while Claude Code costs $18.34, a 17x difference. This tool helps developers choose the most efficient harness for their needs, balancing pass rate, speed, and cost.
Our evaluation shows that the same model can cost 17x more depending on the harness you choose—so we built FrontierHarness Eval to give developers the data they need to make informed, cost-effective decisions.
- vidarh
Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't.
Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops.
Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that.
Claiming on the blog that this gives Kimi Code no home field advantage seems like a dicey assumption. I haven't dug into the newest Kimi Code much, but the older Kimi CLI included several tools that were clearly specifically aimed at working around that behaviour - when I copied their checkpoints and "dmail" mechanism into my own harness, the performance with Kimi improved dramatically, but it made zero difference against Anthropic models.
That doesn't make the data worthless - it's clear you shouldn't use Clade Code to work against Kimi. But it does significantly limit the utility of it.
- joshheitzman
I was excited until I saw cost was only provided as the median. Your provider will bill you for all your tasks and one can get back to that total from the mean by multiplying by the number of tasks. This isn't possible with the median and I suspect the median is likely below the mean so this understates the actual costs.
- nijave
It would be interesting to try to control for system prompt although obviously there's some coupling to what the harness provides and the instructions the model receives on using it
- nsingh2
One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup.
Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.
- markbao
This is great. People always talk about how important the harness is and yet we have so few harness benchmarks. Agree with sibling commenter that ‘harness x model’ is needed.
People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.
Thanks for making this and filling a real gap!