Amazon swaps Claude Sonnet 4.6 for Strands Decider and cuts ticket-labeling costs by 98%
Amazon saves millions by replacing Sonnet 4.6 with Strands Decider
Two Amazon engineers replaced Claude Sonnet 4.6 with Strands Decider, a decision model launched a month earlier, in one of the world's largest internal ticketing systems. The model sorts escalations, capacity requests, alarms, and security findings into 20 categories. It matched Sonnet 4.6's performance at 98% lower cost, saving millions of dollars in token fees each year.
It performed just as well at 98% less cost.
- armcat
About six years ago we built a ticket classification/routing model and pipeline [1]. The initial version was trained using SVMs, it took a couple of minutes to train, and the ticket classification was just a few milliseconds. It's so fascinating to watch how the field moved over to LLMs, and then back to faster/cheaper classifiers again.
[1] "Linnaeus: A highly reusable and adaptable ML based log classification pipeline", https://arxiv.org/abs/2103.06927
NOTE: the work was actually accepted at the International Conference on Software Engineering (ICSE) in 2021.
- Kaedon
At ~$21,700 saved per million tickets, saving millions annually would suggest at least 92 million tickets a year to reach $2 million saved. Is that about right?
I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.
- nlpnerd
Founder of Seldon AI (www.seldon-ai.com) here. We are working on dev tools to help developers with this kind of refactoring and continual production work.
Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one mod […]
- docheinestages
No mention of quality, or at least I didn't see it. What seem to matter nowadays are speed and cost.
- Gasp0de
This article is so confusing. For example, they say they tried the Strands Decider model and dismissed it for Gemma, then they continue to call the model they used Strands Decider? Did I miss something?
- psnehanshu
Ok, but why did they chose Sonnet in the first place? Couldn't they have chosen Haiku or GPT Luna or maybe even some small cheap Chinese model?
- origini
So why did they not even try Haiku 5.5?
Why would Amazon go to another company to let them provide an adjusted decider model?
Amazon already has all these tickets labeld, they could just have build it themselves?
- 3eb7988a1663
Classifying into one of 20 groups? Did anyone try scikit learn?