Robotics Has No Internet to Scrape: The Race to Manufacture Robot Training Data
How to Train Your Robot
AI's recent leaps came from hoarding text and compute, but robots have no equivalent corpus. Teleoperation is expensive and clunky, real-world deployment is scarce, and simulation stumbles on the sim-to-real gap—especially for dexterous manipulation. The field is now improvising: domain randomization, synthetic data, and egocentric video from humans, all in search of a data flywheel that doesn't yet exist.
The biggest challenge we faced was to create environments in simulation diverse enough to capture the physics of the real world. Factors like friction, elasticity and dynamics are incredibly difficult to measure and model for objects as complex as Rubik’s Cubes or robotic hands and we found that domain randomization alone is not enough.
- pakl
LLMs have shown a way to absorb large amounts of textual training data and be useful. For sensory processing you need a different architecture to absorb spatio-temporal streams. Bolting vision onto language models is stretching the tech a bit too thin.
The good news is the data is plentiful — it is all around (the robot). The real world — the robot’s umwelt sensed through its sensors and acted on by its actuators — will provide heaps of data.
Interestingly my colleagues propose that embodied learning will lead to grounding, alignment, and AI safety.
(Worked for 4 years at a robotics + machine learning startup, Director of R&D.)
- alex-moon
This is an excellent write-up and has successfully changed my opinion on something I think will prove critical in the near future: I always laughed at humanoid robots, assuming they were being built for "comfort" reasons. I now realise this is almost certainly the most reliable way to solve the general robot problem - not just for the reasons the OP lays out, but because the whole world is already built for human-shaped agents anyway.
- alexpotato
Sebastian Thrun's talk at Google about winning the DARPA Grand Challenge [0] has a great discussion of training "robots".
Summary:
- the LIDAR had error rates that were not published
- hitting potholes could cause the INS in the car to get confused
- they weren't sure how to fix the errors from the combination of the above
- they ended up just driving the car to generate training data
It worked out well because the computer was predicting if there was an obstacle or not and a split second later the driver of the car would either drive around the real obstacle or the computer would realize that there was no obstacle.
If you've never seen the talk, highly recommend as Thrun is very funny and it's also a time capsule back to when self driving was still a brand new concept.
- ilikerashers
Is the issue also a lack of available sensor data?
Our own nerves produce a huge amount of data. Our eyes and balance also contribute to a very rich picture to enable movement and kinematics. This is all fused into usable feedback.
Arms on fixed axis with disconnected vision feel like we are still very far away from true robot training.
- serpicourge
Here is a good video on the subject by nvidia, very fun prez:
https://www.youtube.com/watch?v=3Y8aq_ofEVs&pp=ygUKbnZpZGlhI...
PS: It's sad that robotic does not seems to get a lot of traction on HN :(