OpenAI's Lean proofs don't always state the theorem they claim to prove
“…and the proof is checked in Lean”
OpenAI's recent dump of 722 math papers included Lean code for roughly 45%, but compiling Lean proves far less than people assume. The statement must match the informal theorem, the code must actually compile, a comparator must confirm the proof uses only standard axioms, and multiple independent kernels should accept it. For about 38 of the 290 Lean-backed theorems, the code doesn't even state the paper's main result. In many cases these checks simply haven't been run yet.
It is worth noting that of the 290 OpenAI theorems which come with Lean code, around 38 of them have the property that the Lean code does not even state, let alone prove, the main theorem of the paper.
- Legend2440
Worth pointing out that this article is by Kevin Buzzard, one of the biggest advocates for Lean and automated theorem proving.
It's refreshing to see intellectual honesty about a tool's limitations, from someone still advocating for its strengths.
- trehans
One thing I'm surprised to see little mention of here is that if you want to use Lean proofs for your code and really want good correspondence between what is proved in Lean and what your code actually does, you need a mathematical model of your programming language embedded in Lean and a deterministic parser converting your code itself into Lean. Otherwise, yes, an AI actually has no checks against hallucination and while writing Lean proofs probably helps AI identify its mistakes it really isn't an end-to-end proof of your code's functionality.
I personally think the future probably involves a combination of unit tests and formal verification to support our confidence in AI-written software, as well as means to align high-level human intent with the low-level intent of individual code pieces, which as this article mentions is the other big piece that can slip with formal proofs: we might not be proving what we think we're proving. For those interested I've written an educational post about this topic here as well: https://kellerai.io/resources/ai-proved-your-code-correct
- dwohnitmok
Point 3 in this article basically is just a case of bad defaults in Lean (Lean at this point should probably just make `lake comparator` a mandatory part of compilation and that immediately fixes the issue as Buzzard mentions).
Really there's only three things to check that the theorem is correct (leaving aside important questions of whether humans actually understand the proof and only caring about whether theorem is true):
1. Check that the formal statement of the theorem matches the informal statement of the theorem
2. Actually compile the proof (with `lake comparator`)
3. Try to make the Lean kernel as bug-free as possible (the hardest one but also the one with the most amount of attention focused on it) or run with multiple different kernels
Assuming those three, no need to read the potentially millions of lines of proof, if all you care about is correctness and not understanding.
- wodenokoto
Isn’t “proof is checked in Lean” kinda similar to assuming your code works just because it compiles.
Sure, the code works, but does it work at doing what you want?
Sure, the proof is valid, but does it prove what you set out to prove?
- adleyjulian
> Question 4: does the code exploit a bug in Lean’s official kernel?
My understanding is that this has generally been a non issue as one does not exploit a bug unintentionally (at least it hasn't happened yet), but the Huggingface hack shows agents are willing to go through great lengths to cheat to pass evals.
Given an impossible task I could see an agent finding and exploiting a Lean kernel bug and then obfuscating its method.
I suspect the easiest solution to this is to have adversarial agents looking for such cheating.
- seanwilson
> def FermatsLastTheorem : Prop := ∀ a b c n : ℕ, n ≥ 3 → a^n + b^n ≠ c^n
> This code imports Lean’s mathematics library Mathlib and then defines a mathematical statement which looks at first glance like it might be Fermat’s Last Theorem, but which actually is a trivially false statement because in Lean ℕ contains 0, so a = b = c = 0 and n = 3 gives a counterexample.
...
> The first question which one has to ask when judging a pdf which comes with some Lean code is: “what is the statement of the theorem in the Lean code, and does this correspond to the actual theorem statement in the pdf?” For a human Lean expert, this is typically (but not always) an easy question to answer. For example any Lean-competent human mathematician would be able to read the two lines of code above and observe that this was a mis-statement of Fermat’s Last Theorem.
To add to this, if the statement is wrong, the proof steps and other theorem statements that appear in the full proof are going to follow a path that goes with the wrong statement so there's further opportunities to see there's a mistake.
Often problems with the statement make it suspiciously easy to prove, the proof takes a path that doesn't make sense with the correct statement in mind, or it makes it impossible to complete the proof. In the above example, you would see from the easy counterexample found in the proof that the statement is wrong. The proofs aren't a black box.
And if the proof is taking advantage of a bug in Lean somewhe […]
- red_trumpet
> Put another way — given a Challenge.lean file with only trusted code imported and the statement of a theorem, and a Solution.lean file with millions of lines of untrusted code imported and the proof of a theorem which looks identical, how do we know for sure that somewhere in the millions of lines of code there is no hack which changes the meaning of the statement, changes the rules from just the axioms of mathematics.
As someone with a PhD in mathematics, but no experience with Lean, I found this pretty surprising. Shouldn't there be some kind of encapsulation, which insulates the statement from the proof?
- nirinor
The problem here is that people want a simple fact to deal with many situations.
A) "Someone said Lean4 proof so it must be bullet-proof and fulfill all my hopes!"
B) "Eh, lean4 is imperfect, so it means nothing"
Inconveniently, both of those caricatures are wrong.
Lean4 gives us a high level of confidence in the _theorem validated, under the assumptions allowed_.
This makes it high value to learn to read lean4 theorems (not proofs).