ProgramBench Vetted: Can AI Rebuild a Program from a Runnable Binary?
ProgramBench Vetted: Reverse Engineering from a Runnable Binary
Vetto releases ProgramBench Vetted, a benchmark of 50 tasks that challenge AI agents to reconstruct a program's source code from a runnable binary and its documentation. The tasks are designed to be fairer and more reliable than the original ProgramBench, with controls against test duplication, environment leaks, and reward hacking. Results show Claude Opus 5 leading with 50% fully resolved tasks, while GPT-5.5 achieves the highest mean reward at 85.2%. The benchmark highlights the difficulty of long-horizon coding and the memorization paradox.
Prior exposure does not imply that a model can retrieve or reconstruct the implementation from memory when asked to move from observed behavior back to code.