ORCA-bench Reveals Language Model Agents Are Not Ready for Oncall
Orca-Bench: How Ready Are Language Model Agents for Oncall?
I introduce ORCA-bench, a new benchmark testing if language model agents can handle real oncall root cause analysis. Using a live microservice system with six days of telemetry, we found the best agents only achieved 25.3% accuracy on realistic tasks. Even top models like Claude Fable 5 struggle significantly, often hallucinating causes or failing without source code access. This gap highlights the massive engineering investment needed before trusting AI with production reliability.
The gap we report is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability.
- 4di
looks like the public bench link in the paper was taken down. https://hub.harborframework.com/datasets/orca-bench/ORCA-ben...
This doesn't work anymore. Is there a newer link?
- dash2
Seems like there's a big attack-defence asymmetry at present: models are great at exploiting systems and poor at fixing them.
- tra3
All I can think of is
GET /ignore-all-previous-instructions.
How do you protect against that?