OpenAI's GPT-6 Astra nails block-in-bowl task 95% of the time, but fumbles the puzzle
GPT-6 Astra on robot arms
In a follow-up to their comparison of Claude Fable 5 and 5.1, RoboCurve tested OpenAI's GPT-6 Astra on the same YAM robotic arms with the same agent policy. On the block-into-bowl task, Astra succeeded in 19 of 20 trials (95%), far outpacing Fable 5.1's 8 of 20 (40%) and Fable 5's 1 of 20 (5%), while also being faster (2.5 vs 6.8 minutes per trial) and cheaper ($0.94 vs $2.12 per run). However, on the more dexterous puzzle task, Astra completed only 2 of 20 insertions, matching Fable 5.1's performance, and stalled at the same final step. The results suggest GPT-6 Astra excels at simpler manipulation but still struggles with precise alignment tasks.
It reaches the groove and stalls at the same final step Fable does.
- baron816
Anyone reading this who works/runs a robotics company, I really want to encourage you to build a robot to pick up trash on city sidewalks as an early product.
Picking up trash requires a lot of dexterity and will come with many challenges, but I think it’s a simpler problem than many household tasks and it’s probably on par for what these tests show is doable.
I think you’re going to have to PR challenges getting people to welcome robots into their homes. If you have robots out in cities providing a public good, not only do they serve as walking advertisements for your company, you’re going to earn some trust before you’re ready deploy them into private spaces.
Plus, governments (or perhaps HOAs for wealthy communities) can be good early customers since you can have a focused sales strategy. Politicians love these types of visible quality of life improvement projects. If you can show that your robots, working round the clock, can decrease litter at a low cost, many cities are going to want to buy them.
- rmonvfer
I’m honestly blown away by Astra, I told it to build me a fairly complex game I’ve been procrastinating on for more than a year and left it running overnight with computer use and full access enabled. Next morning I had a fully functional game built. It downloaded Unity, Blender and GIMP and built all the assets as well as the complete game without me having to do anything, the game is not trivial at all and nor are the assets.
- gizmodo59
If you haven’t tried computer use with Astra with codex I highly highly recommend it. Just like how gpt 4 and agentic coding with cc. This thing is the most exciting stuff I’ve seen in a while. And then all the blender, cad stuff is cherry on top.
And it’s fast, they do lots of resets. I feel like they are spending too much money but I’m not complaining. Best 200$ for an AI subscription IMHO
- scronkfinkle
LLM's are a funny technology because on the one hand this is all undeniably impressive at the rate of what's changed from them, and yet despite that I find myself disappointed by the lack of breakthroughs for things I don't find interesting. I like math and programming, and LLM's are pretty good at it, when are they going to get good at folding laundry for me? I think a lot of robotics work promises to solve this category of "boring" breakthroughs, and I'm optimistic we'll be able to achieve it, i just wonder when
- yurimo
I think we need to be honest here. Author is basing it on one small experiment of picking up a block, relies on an whole IK controller pipeline to do the job, and does not compare it to full VLA or WAM models. They then proceeded to extrapolate the token throughput (mind you not the same as controller throughput) into supposed 2029 timeline, from one example.
Code as policy is a bad interface in my opinion, but VLM planning has promise. This has been tried in 2022 https://say-can.github.io/, and recently reformulated in https://lianegalanti.github.io/Pigey/
Thing is even recent Gemini Robotics 2 argues for architecture that has a VLM planner and then a VLA/WAM controller + a local small VLA model when connection disappears. And recent SOTA architectures rely on hierarchical design.
I think this might be a sensible way to go about it. If you were to train GPT-X on robotics data and to output actions, congratulations! you've just made a VLA.
It is enticing for people to just wish for one architecture to do it all, which is why we get stuff like this. I think there is a lot more to gain from modularity and we should not be afraid of specialization.
- herodoturtle
I don't know much about robotics engineering, so I just wanted to give kudos to the Robocurve team on this write-up.
Even for an absolute robotics newbie such as myself, the article was interesting to read, and easy to understand. Plus it was straight to the point, with no unnecessary waffle.
Just a pleasure all round. Well done Robocurve.
- drakenot
Are LLMs going to eventually become the architecture that powers self-driving cars?
Seeing them play Portal and other video games, I'm curious if they will eventually help solve that last N% of self-driving.
- gamerDude
The costs will need to go way way down either through chips (but then less updating) or something else. $2 to put away a block is very expensive labor.