A cache miss can double an AI request's energy use
Why harnesses matter for energy (and token) efficiency
Neuralwatt's analysis of a month of agentic traffic shows that the client harness—not just the model or GPU—drives inference energy. Uncached input varied from 1.6% to 11.1% across clients, and a mostly uncached request used roughly twice the energy of a fully cached one. Deep in long sessions, a single turn that re-reads most of its prompt costs as much prefill as about 600 normal turns. Most cache breaks came from harnesses editing already-sent prompts.
A normal turn at that depth adds a median of about 370 new tokens, so a single one of these costs as much prefill as roughly 600 ordinary turns.