LLMs and Lean: Making Proof Automation Practical for Dependent Types
We have proof automation now
I have long admired dependently-typed languages like Rocq and Lean for their ability to enforce complex invariants, but the massive time cost of manual proofs has kept them niche. Traditional automation tools like SMT solvers are often unpredictable and require deep expertise to manage. However, Large Language Models now offer a promising new path to automate proof generation, potentially making these powerful systems dramatically more practical for everyday engineering without the usual overhead.
Potentially, LLMs suddenly make dependent-type systems dramatically more practical.
- m1el
Self-insert time.
I spent some time exploring this topic.
Here's my thesis:
Formal verification was expensive. 20x expensive compared to just developing the software, as the author notes.
The cost of finding and developing exploits also was high.
That creates an incentive to put software verification aside, since it solves a relatively small problem, at an extremely high cost.
We've seen how Mythos has found more vulnerabilities than the rest of the security industry combined. (you can argue about the quality and what counts as a vulnerability, but not the point)
So the cost of finding and developing exploits has dropped dramatically.
On the other hand, formal verification is now much easier, since LLMs can automate the proof.
You don't even need to worry about hallucinations, you merely need to trust Lean core.
If the LLM is wrong, the proof will get rejected!
The problem of exploits gets bigger, and the solution of formal verification gets cheaper.
As a result, the needle is now moving in the direction of "more formal verification".
I, personally, think that it is ridiculous that ~none of the software we use is known to work correctly. It just happens to work correctly, most of the time.
My (ambitious) goal is to have a self-hosting, formally verified compiler, which allows proof transfer from source code down to assembly.
I have not achieved that goal yet.
What I have so far:
- one (non-optimized) compiler step which is formally verified
- three simple functions (hex, hex […]
- el_pollo_diablo
I have already written it, and I will write it again: dependent types and total functions do not scale. Maintenance is terrible.
Suppose that you have managed to write a non-trivial piece of software with dependent types encoding all sorts of properties everywhere. The actual computation and proof are intermingled. Think of the author's innocent bound-check proof in the zstd decoder, but across the whole program, with more elaborate properties and longer proofs.
Suddenly, you realize that you need to prove a new property of your program. Can you keep your existing work and build on top of it? In general, no, you have to refine every dependent type everywhere by adding a new conjunct expressing a new invariant, and adapt every proof, as the new property is threaded in the existing program.
That is because dependent types (and other staples of naive approaches to proving program properties, like a unique invariant per loop) structure the program along the wrong dimension: they encourage grouping everything that concerns a value ("put this value in a dependent type that encodes everything known about it") or a program point ("write the precondition for this function as a big conjunction mixing all the concerns"), where it works much better, for long-term maintenance, to structure the development along concerns: computational parts of the program, basic functional properties and absence of UB, termination, other functional properties, security, etc., where each layer builds on to […]
- gz09
Strongly agree with the author here. The future will belong to programming languages that natively embed theorem proofers into their type systems so LLMs can forego a lot of testing by just validating the implementations they write against the specs with formal proofs. Writing formal specs is probably the main skill a programmer in the future will need to get work done.
Verus (https://github.com/verus-lang/verus) is a good start for the rust ecosystem, but it's essentially a standalone language today (with custom syntax and type system).
- Jhsto
As a meta-comment on the topic, something I have noticed is that there still exists confusion what it means to use theorem provers for projects -- the other day I read a tweet from Paradigm, a crypto-VC now seemingly AI-pilled. Some LP of theirs had made a Lean 4 formalization of the Ethereum's virtual machine. The tweet said this would have cost like $150k in API tokens ("would have", as in, I guess they get theirs for free), and took a week of inference time for an LLM to produce. I somehow got distracted to actually take a look at the code, which I found rather light on theorems. Nor did the project make use of Batteries or Mathlib which are arguably the one of the strongest motivation for me personally to use Lean4. That is, I generally rather rely on someone else getting the category theory and algebraic structures right, which then leaves me the proof obligation to show the correspondence with whatever toy I'm working on. Here I'm fine to use LLMs for proof search, very similar to how would I use a SMT solver. But what I have found is that the language models have to be really coerced into using these libraries, because otherwise the models much rather overfit and overclaim a solution with a 3 minute inference task rather than attempt to fulfill the proof obligations over 3 hours. And I feel nauseated when I need to convince the LLM (I use Claude) that filling the proof obligation is for "academic exercise" or because I'm coerced into doing so, because otherwise it will […]
- keithwinstein
This is really cool stuff, and I agree the future is likely to look more like this. I was surprised by the last two paragraphs ("Aside: verified assembly") -- my understanding was that this future is basically already here. I believe agl's colleagues at Google have already deployed some auto-mutated verified assembly versions of some crypto routines, based on the Fiat Crypto + CryptOpt work (https://arxiv.org/pdf/2211.10665), both involving Andres Erbsen who I think is currently at Google before starting a professorship soon. I dunno if this work would count as "cheap" (it looks like Fig. 10 unfolds over the course of a day) but spending a day auto-exploring many verified-correct machine code implementations of the same routine to find the fastest one (which you then ship forever) doesn't seem impractically expensive either.
- dmix
> I've long had a soft spot for dependently-typed languages
Whatever happened to Idris? Is that still being developed?
Edit: Looks like they are working on Idris 2 https://github.com/idris-lang/Idris2
- vessenes
I think it's worth putting a Knuth quote up on your board when you think about verification as an end goal: "Beware of bugs in the above code; I have only proved it correct, not tried it."
Or, as Wolfram proposes in another link on the front page today, the irreducibility of computation means we cannot generally get all bugs out of systems.
To my mind, one of the risks of going all in on verification is that you just move the difficulty to a harder, more abstract layer - the specification. In my experience, this typically looks appealing to a 'type' -- very high IQ, very little real world engineering experience -- it's a dream that hits the inevitable grittiness of the real world and .. the real world keeps winning.