Ferrox: A Rust Inference Engine That Matches llama.cpp on Metal
Building a Rust Inference Engine That Matches Llama.cpp
Ferrox is a pure-Rust inference engine that runs GGUF models on CPU, Metal, or CUDA, with no bindings to llama.cpp. It achieves near-parity or better performance on Apple Silicon, hitting 28.3 tok/s on Llama-3.1-8B Q4_K_M (vs llama.cpp's 27.6) and 196.1 tok/s on Qwen2.5-0.5B (vs 132.8). Key optimizations include fused dequantization and architecture-specific Metal kernels. The project provides an OpenAI-compatible server and a CLI, with benchmark receipts for reproducibility.
That is the moment the engine stopped feeling like an academic exercise.
- noman-land
At this point all vibe coded projects are an attack vector and should be avoided.
There's simply no way to easily tell by traditional means if they were made by a curious amateur or a malicious acter.
- satvikpendem
I was looking for this and to even build something like this in order to have a pure Rust stack for my backend, as well as embed local inference on edge like on mobile, but llama.cpp is just too far advanced to catch up, it seems. There are some but I'm not sure how good they are.
- ubedan
Wonderful... I'm so happy to see a Rust version of llama.cpp.
The true value of this will be proven over time with wide use and as PRs are merged.
Do you have a feel if you'll try to drive this to stay feature parity with llama.cpp, or are you willing to diverge with new features like NVME/SSD MoE weight streaming etc.