Five Qwen3.8-27B models beat Fable 5 on LiveCodeBench Hard
Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard
A Memdoor workflow rebuilds GVS5H's five-agent harness (planner, two ideators, solver, manager, plus a repair worker) on Qwen3.8-27B, with agents sharing only files on disk and a shell script deciding success. Over five passes on the first ten LiveCodeBench Hard problems, it scored a mean of 9.8/10, versus 9.6 for the authors' own harness and 8.0 for Fable 5 alone. All 50 runs are published; contamination can't be ruled out.
The problem text was the mismatch: our first version wrote its own task file around each problem, and the 27B planned worse from it than from the authors' exact LiveCodeBench text.
- fedpost
We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
- roenxi
That's pretty interesting, because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine [0]. Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
[0] https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw... suggests 16GB VRAM, 32 GB system.
- xlayn
Remember those gpus into chips https://inference-docs.cerebras.ai/models/qwen-3.8-27b , where you have this chip that implements a model that run at those extremely high speeds... the ones that people ask "why do we need that"... if this is true... one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet...
Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...
- mkl
GVS5H, which this is based on, was discussed here a week ago: https://news.ycombinator.com/item?id=50228167
- yurimo
There is something to say about the power of brute force approaches. Right now scaling multi agent is seemingly what is becoming the next important axis, though very strongly constrained by quality of communication protocol and compute. Even if one model is not strong enough individually, more models and more attempts as well as differing opinions may nudge the models into a better solution basins.
Kind of an extension of the effect where you talk to a model, it is super confident (but wrong), you correct it not directly but perhaps with a question and asking it to clarify. This then in turn leads to a better approach overall.
- zihotki
I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list". And LiveCodeBench Hard is a several years old benchmark. Could be very saturated
- vrinsd
Maybe slightly off-topic from the main article but is there a good resource or site that clearly helps understand which of the "open" models are good for programming tasks, like C, C++, Python, Java?
I've looked and there are tons of sites that aggregate models vs. benchmarks but I'm not sure how much of that is trust worthy.
- bad_haircut72
This is probably not a novel idea but I wonder how it would work if you just stacked a bunch of smaller models on top of eachother and used that as the base to train a much much larger model. Maybe more efficient than starting from scratch?