RTK promised 90% token savings, but our $1,500 benchmark found none
RTK reports token savings, but our cost benchmarks disagree
RTK, a popular tool with 79k GitHub stars, compresses terminal output before AI agents read it. We spent over $1,500 testing it on Terminal-Bench 2.1 with Claude Code and OpenCode. RTK's reported 349 million tokens saved didn't translate to lower costs: Fable's savings came from one task, while DeepSeek's costs rose 17% on average. RTK can add turns, and its gain metric counts removed output, not money saved.
Large reported token savings did not mean cheaper tasks.
- aeneas_ory
All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md.
What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly. Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.
Why does it work you may ask? Well, LLMs basically brute force words/phrases and pipe that into find/grep/pgrep/whatever (or as recently discussed here write a python script for it - https://news.ycombinator.com/item?id=49654229). Semantic search looks for similarities so you have to do less brute forcing. Comes of course at the cost of indexing everything first.
You can find the project here: https://github.com/ory/lumen
- ProjectBarks
It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves?
My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark.
Here are other cases demonstrating the exact same issues with these kinds of tools:
https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-...
https://brandonbarker.me/writing/headroom-fewer-tokens-bigge...
- oefrha
It's pretty damn obvious to anyone who ever bothered to look at rtk gain output, no benchmark needed at all. Agent runs
rtk command-that-prints-100k-tokens | tail -5
costs 5 lines, maybe 100 tokens without rtk, but rtk will report 100k savings. Of course it doesn't know about that tail -5.
Worse, since rtk defaults to persisting that savings stat, it breaks sandboxing. Prefixing with rtk leads to random auto-mode denials from time to time too (this is independent of disabling savings stat persistence).
Honestly have no idea why anyone who knows the first thing about CLIs would take rtk gain seriously. I guess clueless vibe coders who has hardly ever worked in a terminal before will look at the stat and feel good about it?
That said, rtk is still mildly useful for compressing repeated test run outputs and stuff, but you should only ever use it on whitelisted commands; wrapping everything like they suggest you to do is just stupid.
- kriskrunch
Open question, how does this instruction in agent-rules.md look?
"Cap large/unknown command output: `COMMAND 2>&1 | head -c 4000`. Never stream full logs, tests, or large files."
I use that instead of RTK. Empirically, I found RTK makes my agents run longer to complete similar tasks.
Ponytail and Caveman seem to help somewhat.
- stephantul
I think anyone who is even a little bit realistic knows that most technologies overclaim, or evaluate under very favorable conditions.
This is not a good thing of course, but I also feel that acting surprised that this is going on is a little unnecessary.
Having said that: most tools are not helpful