LeWM's predictions already match perfect physics — the real gains are in planning
After 600 experiments, here's what I discovered about JEPA world models
After 600 experiments on LeWM, a JEPA world model benchmarked on Push-T, Haddy Alchaer found its learned predictions plan as well as the simulator's exact physics over its trained paths. The bigger wins came from how the model is used: chaining 25-step subgoal searches and planning through a memory graph of demonstrations lifted long-horizon success from 22% to 85% without retraining. A cost on jumps between demonstrations, and filling memory gaps with rotated or mirrored segments, extended it to harder goals.
Along the kinds of paths it was trained on, LeWM's predictions are already as good as perfect physics for planning.
- robotresearcher
There’s a repeated theme here that I don’t understand. For example:
> (3) That's where I decided to build a harder test, where each task takes its start from one demonstration and its goal from another
> (4) I also built an interactive demo where I could place the block anywhere and pick any goal. [but]
The goals in the harder test are frames from expert demonstrations
Why lean so hard on the demonstration examples? Surely the actual task here is to move between any two arbitrary poses.
So if the demonstration set is statistically strange in any way, you keep banging into that. And even if you get great results, you still haven’t shown that it generalizes to the generic system. And the one thing you know 100% about a physical robot system is that you will never encounter exactly the same situation twice.
Is there actually a reason to pick eval examples from the demonstrations instead of independent samples from the underlying task?
I’m pretty sure that the T-shoving problem doesn’t have goal reachability issues. And if it did, test 4 could run into them since the start pose is arbitrary.
- hazrmard
JEPA is very similar to Model Predictive Control. Here, you swap the model of the system with black-box simulator (a neural net), apply some version of dynamic programming / Bellman equation to find the optimal action over planning horizon.
In the planning stage, by factoring out the optimization step from the rollout step, you can free the model to focus on learning the dynamics in latent space. (The other case, where the rollout+optimization happen in the learning loop, just becomes model-free RL.)
My background straddles classical & data-driven controls. This was the first similarity that popped out to me when I first started learning about world models.
Is JEPA just MPC but where the "M" is a latent-space model?
- reasonabl_human
Hands-on learning is an awesome thing - especially in the age of AI!
But maybe worth reframing away from “discovering” - hierarchical planning, sub-goals, action chunking etc. are well understood - this is more of a… transcript of your personal learning journey?
Which is still cool to see. But I read through expecting there to be some kind of novel results. We probably need to categorize these kinds of posts a bit better.
- halcdev
Let me know if you have any questions!
- cheesecompiler
The theme is really hard to read. Dont mean to be a dick, I love the article. I tried a few browsers and my eyes really strained.