OpenAI Retracts Recommendation for SWE-Bench Pro Due to Benchmark Flaws

OpenAI no longer recommends SWE-Bench Pro

We recently audited SWE-Bench Pro, a coding benchmark we previously endorsed, and discovered significant design flaws affecting over 30% of its tasks. Issues like overly strict tests and underspecified prompts mean the evaluation no longer provides a reliable signal of model capabilities. Consequently, we are retracting our earlier recommendation and urging the community to exercise caution when interpreting results from this dataset.

"Ultimately, an eval should provide meaningful signal through benchmarks that are hard to game, easy to trust, and genuinely reflective of model capability or alignment."