Pan: A General Goal-Conditioned Minecraft Model for Autonomous Robotics
At Pantograph, we trained Pan, a 4B parameter model, to act autonomously in Minecraft by pretraining on internet-scale videos. By treating video as reinforcement learning trajectories and using hindsight relabeling, we learned goal-directed behavior during pretraining rather than post-training. This approach allows Pan to generalize to unseen environments and achieve complex goals like building structures and fighting mobs far better than existing models like STEVE-1.
If we can do reinforcement learning at scale on video, we may be able to produce models that are superhuman in many more areas all at once.
- agajews
Hey everyone! I'm Alex, one of the founders of Pantograph. We've spent the last six months building a pretty smart Minecraft model, coming soon to a server near you!
We trained it on about 500k hours of Minecraft videos, and it learned how to fight creepers, build walls and other structures, and explore to find visual goals.
We're considering putting up a public API for larger models like this one, let us know if you'd like to be able to put Pan in your own server :)
What's most interesting about the model isn't the performance that it gets in Minecraft, but how general the method is. When we scale it up, it should be able to act in any kind of video game, as well as robots in the real world (which are really just another video game).
- Onavo
I disagree with the approach. It's a good approach for limited domain problems, but not for general purpose. Take something like this where you will need to be able to refer to wikis and research and ask questions on Reddit and Discord to optimize playthrough, none of the goal conditioning will be useful.
https://www.feed-the-beast.com/modpacks/125-ftb-evolution
I think a properly fine-tuned VLA with access to tool calls can scale way better.
- xnx
Reminds me of Google's AI Dreamer that could mine diamonds in Minecraft (https://www.scientificamerican.com/article/google-deepmind-t...) or OpenAI training on video to play Minecraft (https://openai.com/index/vpt/)