Pan: A General Goal-Conditioned Minecraft Model for Autonomous Robotics

A General Goal-Conditioned Minecraft Model

At Pantograph, we trained Pan, a 4B parameter model, to act autonomously in Minecraft by pretraining on internet-scale videos. By treating video as reinforcement learning trajectories and using hindsight relabeling, we learned goal-directed behavior during pretraining rather than post-training. This approach allows Pan to generalize to unseen environments and achieve complex goals like building structures and fighting mobs far better than existing models like STEVE-1.

"If we can do reinforcement learning at scale on video, we may be able to produce models that are superhuman in many more areas all at once."