Why we build our own C and C++ inference engines
Why we write our own C and C++ inference engines
LocalAI's team explains why they sometimes write custom C/C++ ports of AI models instead of wrapping existing engines. The payoff: a 66 MiB binary that matches vLLM's throughput and memory footprint, and a CPU port that beats PyTorch 1.31x on depth estimation. But the real gate is parity—outputs must match the reference exactly—and the cost is heavy maintenance, especially on GPU kernels.
A port that is fast and slightly wrong is worthless, and without a per-component gate you find out it is wrong months later.
- stephbook
Should have started with writing your own blog posts.
- dennis16384
I had a similar success with Model2Vec static embedder and NER inference (both GGUF, compiled for WASM), ported to plain C from ONNX Runtime.
Wasm size from 30Mb to 300kb and 1.5x speedup. It's definitely worth it for performance or distribution size.
- aabdi
I don’t think it would be surprising that people want to write their own kernels.
A big problem with the existing engines like llama or sd is that they don’t support optimal graph compilation. Usually this means about a real 2 or 3x multiplier loss relative to optimal. Cuda graphs do okay but they still leave a lot on the floor
It’s usually worth it to optimize in that context if you are willing to peer into the mechanics.
Of course that’s expensive. You need to know how to appropriately pipeline and merge your kernels.
- scottcodie
I did took a native c++ approach when writing a relational transformers engine (RelativeDB). My journey was pytorch -> c++ -> Triton (lang). While C++ was more performant than Triton, I couldn't afford to optimize on every gpu. I just accepted the ~15% throughput loss for my cloud service, which honestly wasn't bad for the amount of flexibility I got out of it.
But the cpp port of vllm looks great, that'd be great if you'll maintain that. I hit the same limitations with vllm.
- piterrro
Could this vllm port be faster to install? Im starting gpu machine multiple times a day and it takes 5 minutes to set vllm up. If Inise this port that time is minimized?