Migrating a Production AI Agent to GPT-5.6: 2.2x Faster and 27% Cheaper
Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper
We switched our production agent from Claude Opus to GPT-5.6 Sol, achieving 2.2x faster builds at 27% lower cost. This migration required fixing our evaluation harness and adapting to GPT-5.6's unique behaviors, such as its tendency to fill all tool parameters and its distinct prompt caching mechanics. By redesigning our tool schemas and cache strategy, we unlocked significant performance gains while maintaining high-quality output.
Your harness is tuned to your incumbent model, and you don't know it.
- kristianp
> Numbers like that buy a model a real migration effort.
Such a silly choice of words. I wish the human directing the LLM writing the article put some effort into rewriting the worst examples of LLM style.
> But it did extremely well, and the promise was immediate and specific: builds finishing in less than half the wall-clock time, at 27% lower cost, scoring at or above our incumbent on completed work.
The way the LLMs write (Claude perhaps?) With short phrases separated by colons, commas or full stops, is so poor and frustrating.
There some good insights behind this article, so it's worth reading, for example below, but it isn't easy to read.
> Earlier GPT models cached implicitly on partial prefix matches, which gave decent hit rates for free. GPT-5.6 dropped partial-prefix matching:
- thiagoperes
We run a lot of varied, tiny, simple workflows that were previously running on 5.4-nano and mini. We transitioned them to 5.6 and noticed exactly this range of improvement across the board. In a few cases, we had improvements in classification.
I think a lot of people miss that for many companies, a model upgrade like this is basically a one liner.
Even if you have an amazing model router architecture (which we do for our golden flows), it’s just not worth it. Not to mention reliability and so on
- blfr
> Ploy’s agent builds and edits real marketing websites. It plans a page, reads the codebase, writes components, generates imagery, screenshots its own work, and decides when it’s done. That job description sets a very high bar for a model, and we test every frontier release against it. For the four months Opus held the default slot (first Opus 4.7, then 4.8), nothing we tested beat it.
Well, unlike OP I haven't run a rigorous test, but I still would expect Fable to be significantly better at building marketing websites than Opus. It sure is way better at building decks.
- znnajdla
My experience mirrors this: services like OpenRouter that promise “failover” are pretty much useless except for sandbox testing because models in production are not really interchangeable. Any production harness doing serious agentic work in production is dependent on more model-specific quirks than you would expect. And even if another model works without errors, performance and efficiency is a whole different story. Even the system prompt can and should be tuned to a model’s preferred speaking style, for example <xml tags> for Claude-like models because they were trained on it, while other models do better with other delimiters. Think of the whole harness, prompt, and model as one system, not really with modular parts that can be swapped out if you care about optimal performance.
- throwa356262
As of today, Ploy’s agent runs on GPT-5.6 Sol, the flagship tier of the model family OpenAI released this morning.
Wait a moment, did they make the switch based on half a days of playing with Sol? Are these companies ran by teenagers?
- SwtCyber
Its ironic that under an article with a ton of deep infrastructure insights half the comments are crying about the "forced writing style". What does it matter if claude helped the author clean up the text when inside is a ready-to-use blueprint on how to save 30% of the api budget and fix empty file reads?
- lcampbell
> The fix that worked is a schema transform at the provider boundary. For OpenAI-family models only, we rewrite every optional property to be required but nullable, using anyOf: [T, null], which gives the model an explicit way to say “not using this.”
I admit, I've only used a bastardized form of MCP, but this smells... wrong? It's not clear to me why the Typescript type definitions would have any influence on (what I presume is) JSONSchema being sent from the agent to the inference backend as part of the completion request. The MCP specification (which the OpenAI backend might not use, I don't know) has an explicit field to signify "optional" parameters in the JSONSchema; my read on this is there's a bug somewhere between the Typescript layer(??) and the generated tool description which is actually sent to the inference backend.
It's possible the inference backend has changed from "generate valid tool responses" to "generate valid tool responses according to the JSON schema [where no parameters are optional]" but it's impossible to tell without seeing the actual requests sent to the inference backend (which I didn't see in TFA).
- arikrahman
Migrating my workflow to Reasonix with cache hits on Deepseek make requests practically free, and that's on unsubsidized American providers.