AI Coding Agents Fail on Real Enterprise Codebases, Top Model Resolves Only 38.8% of Tasks
Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
Real-SWE, a new benchmark from Specific Labs, evaluates frontier AI models on private, production codebases licensed from real companies. Tasks include fixing billing, tax calculation, and customer migrations—work with business consequences. The best performer, Fable 5.1 with Claude Code, resolves only 38.8% of tasks, and six of ten tasks have resolution rates below 15%. Missed requirements are the most common failure, highlighting that agents struggle with company-specific coding patterns and context.
99% of tokens in real-world enterprises are hidden away from the frontier models.
- springtimesun
I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don’t go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR.
The thing that takes the most time is finding the examples. In my real dev flow it’s rarely ticket -> PR -> merge, things bounce around a lot more. So, even though the stated goal is to get away from one shots, that is basically the environment you have to set or else test for specific other outcomes (e.g. agent stopped and raised a question when it realized x).
It takes time to do, but I would really recommend it. Now I can push new open models through the batteries and see how they line up to past ones in a few days (I run them locally, it’s slow). It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic (these still run at temp 1, heuristic is the correct way to think about outcomes IMO). It grounds it in your actual code and problem space.
My takeaway from my testing: in Rails or front end codebases, most models I test are competent and with a human in the loop they would accomplish their goal of getting to a mergeable PR. They are not as good as Claude and since I pay subsidized rates via subscription Claude still gets first pass. They are very worthwhile to layer in as review […]
- glub
I'm surprised Sol and Astra are leading "Unverified assumption" metric and Fable is better there.
I run Fable as my main model with Sol as advisor that watches every turn. Fable likes to throw around assumptions that it didn't check that are simply false, and Sol always goes to actually verify them and then alert Fable it's assuming things.
I've tried reversing this pairing with Fable as advisor. It'll just sit there going "sounds good"
- prometheus1992
Does this mean they ended up sharing those private codebases with OAI, Anthropic etc? Also, the ~30% number tracks with my experience. I thought I was going insane for expecting too much from the models but they are still bad, including astra. This morning it messed something pretty trivial while fixing an issue which I was shocked to see. Also2, benchmarks don't mean much these days.
- lmeyerov
My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point.
One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.
- jstummbillig
I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.
- chandureddyvari
It also depends on your skills and tooling (test execution and verification- agent browser, functional/unit etc)
GPT 5.6 Sol lagging behind Kimi, GLM 5.3 is surprising to me.
IMO Fable 5.1 ~ Astra > GPT 5.6 Sol > Opus.
- matheusmoreira
I used a similar methodology. Code review is my most requested action, so I used blind code review results to compare the frontier AIs.
Even posted an article about it:
https://www.matheusmoreira.com/articles/code-reviewing-lone-...
Unlike TFA, the lone lisp code is public. I suppose the models could have been trained on my codebase. Still, I think it produced some interesting results.
Took months and loads and loads of tokens to do this, so I'm not gonna repeat this study as new models come out. It did anchor all of my future expectations, though. OpenAI is winning as far as I'm concerned, and their cybersecurity program is the only remaining pain point.
- skhameneh
Some things in this seem reasonable, but others just don’t make sense and there’s crucial details missing (like reasoning levels and what harness was used).
For example, I found Kimi K3 to use more tokens than some other models, which caused it to cost twice as much purely because of the token volume. This experience lines up with ArtificialAnalysis’s benchmarks, but not these.
There’s a number of other comparisons here that don’t match up with my experience or other benchmarks. By many accounts, this is the outlier.
I could attribute the differences to harnesses used or something like reasoning levels, but none of those details are published.
While this seems interesting, I can’t take this seriously.
Correction: The harnesses are listed as a column, I missed that. My other concerns and questions still remain, it’s unclear why some of their results are the outlier that does not match my experience, ArtificalAnalysis’s benchmarks, or some of the experiences of others commenting.