HEIR compiler: 4,000x slowdown for encrypted ML inference, but GPUs cut it to 50x

Updates on HEIR, the Homomorphic Encryption Compiler Project

Jeremy Kun, a Google engineer, provides an honest, technical update on HEIR, the homomorphic encryption compiler project. He shares concrete runtime benchmarks from a demo repo: a credit card fraud detector takes 2 seconds for encrypted inference versus 0.5 ms for cleartext, a 4,000x slowdown, while a recommender model takes 5 minutes on CPU but ~500 ms on a GPU (50x slowdown). He discusses the compiler's MLIR-based design, its current limitations (manual annotations, incomplete operator coverage), and his perspective on HE's 'killer app', arguing that private LLM inference is still far from practical and that lower-hanging fruit should be pursued.