Why AI Agents Fail at Computer Use: The Interface Problem
You can't solve computer use by ignoring the interface
Current AI agents struggle with real-world tasks because they bypass user interfaces to hack APIs instead of interacting naturally. This approach wastes compute on perception rather than reasoning. At Steelman Labs, we argue that scaling models is a dead end. True progress requires separating planning from execution to achieve human-level motor control and reaction times.
We use trillion-scale reasoners to work around clicks.
- juancn
You lost me at the annoying trail on mouse movement.
You advocate for good UI and immediately make some crap that makes it hard to read/use your site. It's kind of ironic.
Please stop messing with: selection, mouse cursor, copy/paste, etc.
- drob518
The article makes the point that on some benchmarks the AIs worked around the UI and then says that sometimes APIs won’t be available. The unwritten implication is that then we’ll be in trouble. But will we? If an API doesn’t exist, is the AI still able to perform the task? That’s not covered in any detail, nor is there any direct comparison between the frontier AI’s performance on these tasks and the model being sold by the company. At some level, I don’t fault the AI for taking the API path if it’s available. In fact, I’m impressed that it found the API and used it correctly. That doesn’t seem like an argument that the sky is falling.
- gandreani
Meta: I like that the first comments are not about the model at all but the website's mouse trail. One is in favor and one is against. Sums up society pretty well hahaha.
On the model. Really cool that it can act every 70ms. That's more than 10x faster than TTFT in most inference APIs. It HAS to be a local model. No idea which family or which one it is or what hardware it runs in. These would be nice details to tease!
- kvark
We have 2 paths here:
- either evolve our UI systems for computers and let them operate efficiently, i.e. converting everything to API calls
. Frontier labs are pushing for this, starting with the Chrome plugin that Claude Cowork mandates.
- or make AI systems use the screen like humans do - what the article is arguing for
Both of these paths are possible. One of them is much more efficient than the other in the long term. Another one may be inevitable.
- nvch
Looking how much agents like to use and push to have functional a11y trees, we may accidentally solve accessibility as well