39.5% of 'Correct' LLM-Generated GPU Kernels Are Broken: A New Verifier Exposes the Gap

A Contract-Grade Verifier for LLM-Generated GPU Kernels

Systems that generate GPU kernels with language models claim high correctness, but their tests are loose—running a few random inputs at one shape and checking for approximate matches. A new contract-grade verifier applies twelve adversarial gates, several tolerance-free, to audit 2,638 kernels previously accepted by a public system. It finds 39.5% broken beyond any tolerance argument and 62.1% with at least one violation. The verifier also validates a native Blackwell backward for the gated-linear-recurrence family, demonstrating that the field's correctness signal is far weaker than reported numbers suggest.