Coding Agents Think Ahead: Unveiling Their Latent Programming Horizons
Coding agents think ahead of time
We discovered that coding agents internally represent program properties long before making edits. By analyzing hidden states, we can predict if code will pass tests or introduce regressions up to 25 steps in advance. This latent programming horizon reveals that these agents are reasoning about future outcomes while we watch them work, opening new doors for understanding AI behavior.
These representations run ahead of the agent's own edits, predicting the outcome of future changes up to roughly 25 steps in advance.
- HarHarVeryFunny
This feels to me more like incremental belief building than "thinking ahead of time" (which is not what the paper is claiming).
The model only has partial observability of the program it is working on (whatever tool call outputs are present in the context), as well as the trajectory of actions it has taken, and from this is building up some internal beliefs about the program - the probes used were looking for pretty crude things like "is this program well-formed" and "is this program correct (will it pass tests)".
The paper says that these program "properties" (beliefs) predict future state of the program up to 25 "steps" ahead, but given the setup this seems to be expected. An agent is trying to fix a program and/or maintain it in a working state, so it doesn't seem surprising that current well-formedness and correctness persist into the future, or that the model is correctly "optimistic" about the outcome of the next action it is planning/predicting.
This incremental belief building from partial observability reminds me of the ability of LLMs to predict valid chess moves when only given a truncated history of the games moves so far (e.g. last 20 moves, not all moves back to start of the game).
- energy123
Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.
- x312
It's been known for several years that LLM activations encode future tokens ahead of time (e.g. https://arxiv.org/abs/2404.00859).
But this has only been shown on simple tasks, so I think this paper is still quite neat. The interesting thing is that they show "future horizon length" varies across models.
- subygan
This is true of truly skillful people in their craft as well.
I know people, who initialize all required variables and write the logic after. which used to feel bonkers to me until I realized, they've done enough practice and memorization to be able to figure what they would need 10 steps down the line.
this does show that, models have a better model of the task and the expected end state.
- bluegatty
The paper is saying 'context is context'?
That after a model has context about a project, the probes indicate a state that validates that?
Seems that the paper is highlighting the very nature of what LLMs are and what we expect them to be?
And that there is no 'thinking' here, it's just the state of the model?