I let an LLM control a robot arm overnight — and it stacked blocks by morning
Getting out of the way: my robotics crash course
A software engineer with 25 years of experience bought a preassembled LeRobot SO-101 arm and a webcam, then handed control to LLMs. After calibration and spatial mapping failed, he removed himself from the loop and left Claude with a simple goal: pick up red, blue, and purple blocks and place them in a taped area. He woke to find all three blocks moved. A time-lapse from screenshots captured the process.
I woke up to find all the blocks in the taped-off placement area: they weren't perfectly aligned or neatly placed but they were all there!
- WoodenChair
I thought this was going to be a post about learning robotics.
It seems more like a post about hooking up an LLM to a pre-made robotic arm.
While that's interesting, I wouldn't have labeled the post "my robotics crash course."
Unfortunately, I think this is in some sense another example of LLMs substituting for actual learning. While I'm sure the author is learning something about robotics from doing these experiments, I doubt it's as much as he would have gotten from say reading a couple chapters of an introductory robotics for dummies book.
- jvanderbot
First law of calibration: Make sure you do an error analysis.
Finding the table surface is pretty useless using a top-down view, even with April tags, because the range error to an april tag is much more than the bearing (pixel) error to the april tag. You basically have trouble observing the thing you're trying to measure.
If you do this again, ask your agent to conduct this analysis and make sure your desired calibration variables are observable with small error. A second camera from a 45deg angle or even on the table would go a lot further, but then of course other things become unobservable.
Nice workaround using a proxy for force sensing to get touch info, however. And neat project overall!
- pj_mukh
Shoutout to people who don't overthink blog setups. "thisismypersonalblog.com" AND GO. It'll do the job.
On a technical note, your setup is not far from where professional setups are headed [1]. Astra is really doing an end-run around (for now) research Robotics setups.
[1]: https://x.com/ihorbeaver/status/2104646736652447854?s=20