How Pi Prevents Context Overflow with Smart Compaction
How Compaction Works in Pi
Coding agents like Pi, Claude Code, and Codex hit context window limits as conversations grow. Pi's solution: compaction, which summarizes older turns into a structured brief while keeping recent messages intact. This post explains the mechanics—from the compaction prompt that acts like a shift handoff to the impact on prompt caching—and notes that Pi's extensibility lets users swap in custom compaction logic.
The ideal outcome of a good summarization for a coding agent is like a handoff briefing from one shift to the next.
- kierangill
Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages.
For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration.
Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations lead to more frustrating future chats because the LLM misses intent and or context. (Or, the presence of paragraphs and paragraphs of LLM output makes the next token predictor dumber? Unsure.)
- novaRom
Compaction is painful if you run just one local LLM, the best way to avoid it is to keep context as small as possible.
One trick I find useful is to have one model with two KV caches running and while first cache has produced tokens, second cache immediately summarizes them during input tokens are being generated (tools time), then harness switches to the second KV cache which takes newly produced input tokens while KV in first cache is getting replaced with compacted summary tokens. This is a kind of ping pong, so we trade more space for less time. Still experimenting but it looks it works, and nice bonus it improves GPU utilization. Btw I have my own harness and model serving code, but it can be easily implemented in any other harness and model server.
- skeledrew
I think the way prompt caching works really discourages more creative compaction techniques. Like perhaps some kind of heuristic progressive compaction that replaces tool results and thinking traces after use with pointers could potentially keep the model smart for much longer, but that'd mean breaking cache every turn, and possibly even within a turn, seriously driving up cost.
- damsta
I don't like any of current solutions when it comes to compaction. I'd love to have a way to say what exactly should be summarized, because most of the time I just need to compact some noisy MCP tool calls, test runs and things like that. Just let me pick what should be summarized and keep the rest as is.
- jakswa
OMP changed the default compaction to images! Kinda nuts to read about. Saves the generation cost of the traditional compaction step and writes the context as tiny text to an image, if I was following correctly.