Reame - CPU-first LLM inference server that accelerates with use
Show HN: Reame – a CPU inference server that gets faster as it runs
Reame is a lean, fully-tested LLM inference server built on llama.cpp, specifically designed to maximize performance on affordable CPU hardware like shared vCPUs and free tier ARM boxes. Unlike traditional servers, Reame treats low-cost CPUs as first-class citizens by utilizing a persistent disk-based KV cache and self-regulating speculation to ensure the same computation is never performed twice. It excels in narrow, repetitive workloads such as document extraction and batch processing, offering an OpenAI-compatible API that allows developers to run private, sovereign AI models at zero marginal cost after the initial request.
On a CPU, never compute the same thing twice.
- JackC
> What Reame is NOT for — said plainly, because trust is built here
It is so weird (or, used to be) to see an LLM's internal thought process pop up this way. Like imagine how strange it would be to read human writing that accidentally included thoughts undercutting the ongoing sentence. It's the moment you know that nothing you're reading has necessarily been seen by a human before or relates to reality.
- ComputerGuru
Seems the entire repo and documentation is all ai-generated?
- JPLeRouzic
Thanks for sharing your work!
Please how to select the model?
I downloaded tinyLlama, put it in ./models, changed reame.conf but I get:
(No such file or directory)
Otherwise putting the model in /opt does not please me much, I fear to forget a model is there, if it is in reame folder its much easier to notice and manage.