The One-Step Trap: Why Simple AI Models Fail at Long-Term Prediction
The One-Step Trap (In AI Research)
I argue that relying on one-step predictions to model the world is a dangerous mistake in AI research. While appealing, these models fail because small errors compound rapidly over time, and calculating long-term outcomes becomes computationally impossible. Instead of iterating simple steps, we must build temporally abstract models using options and GVFs to truly understand complex futures.
The bottom line is that one-step models of the world are hopeless, yet extremely appealing, and are widely used in POMDPs, Bayesian analyses, control theory, and in compression theories of AI.
- ssivark
Ha, interesting. I wasn't aware of Sutton's blog post, but if I might make a shameless plug, we demonstrated [1] exactly this problem (see section 4.4.3), and how multi-step world models (using diffusion models as the substrate) could be one potential answer.
Since then, I have come to like temporally-abstract models more and more. Rolling out in time -- either step-by-step or many steps at once -- suffers from the tyranny of the specific. For long horizon planning with agents, I care (often only approximately) about where I can end up, and seldom about exactly when I end up there. Successor features, GVFs, Forward-Backward representations, and the like seem like they have an elegant approach for structuring thinking at a "high level", instead of generating exponentially large search trees by rolling out microscopic world models.
[1] https://arxiv.org/abs/2410.05364 (funnily, from around the same time / few months after Sutton's blog post)
- mxwsn
This is the same reasoning behind why Yann Lecun thought test-time scaling would not work for LLMs: compounding error.
Instead, the more tokens LLMs use, the better their performance on many tasks. LLMs can self-correct, evidenced by the power of getting models to question themselves by emitting "Wait," in S1. https://arxiv.org/abs/2501.19393
- fny
I'm not sure I follow what one step means exactly. Aren't all models some f(x) = y? Is the suggestion instead that we should be doing f(x) = g(h(x)) = y?
What would the difference be?