Self-hosted inference orchestrators compared: LocalAI, exo, GPUStack, vLLM
A September 2026 survey of self-hosted inference orchestrators for OpenAI-compatible endpoints. It compares Ollama, llama.cpp, vLLM, LiteLLM, LocalAI, exo, Xinference, GPUStack, NVIDIA Dynamo, llm-d, SkyPilot, dstack, and CoderAI across modalities, multi-machine support, auto-discovery, cache-aware routing, ops consoles, cloud burst, non-LLM fan-out, training, Kubernetes, platforms, and signed images. The author, who built CoderAI, gives honest recommendations for each situation.
The parts nobody else has are the three-tier escalation — a model runs on your card, then on a machine you own, then on a RunPod GPU rented by the second with a price cap and a budget, chosen per model and switchable to "only when busy" — the fan-out of image, video, embedding, speech, transcription and OCR requests over every machine that has the model, a video pipeline relayed part by part, and LoRA/QLoRA training data-parallel across nodes from the same server.