Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase

Benchmarking coding agents on Databricks' multi-million line codebase

We benchmarked coding agents on our massive Databricks codebase to find the best balance of cost and performance. Our analysis revealed that open models like GLM 5.2 now rival top-tier closed models, while token prices poorly predict actual task costs. We discovered that the harness used significantly impacts efficiency, proving that a mix of tools is essential for frontier performance.

"The token price of a model is a poor indicator of actual costs incurred on end-to-end tasks."