Spinf-serve - Bulk classification and decision inference on one GPU
Show HN: Spinf-serve – inference server for bulk scoring and decisions
Spinf-serve is an inference server that reads content once and answers many questions in a single pass, returning probabilities instead of generated text. Built for bulk classification and decision models, it runs on one 24 GB NVIDIA GPU and serves an OpenAI-compatible API. It solves the inefficiency of chat-based engines that re-read content for every question, making large-scale scoring fast and cost-effective. With prompt packs, extraction, and batch invariance, it's ideal for moderation, triage, and market analysis. Apache 2.0 licensed, it's available as a pip package or Docker image.
The content is read once; every question is a branch of the same pass and costs its own tokens only.
- Arkada
Loving that this comes out of a market news pipeline. We have something similar but smaller scale in our trading bot app where we are trying to score incoming crypto/macroeconomic news based on fixed questions like importance, direction, and coins affected.
Doing that via standard chat API is just a bit too much if all you need is quick yes/no or scoring. Adding `floor_p` for the user to see the baseline bias of the model when given nothing is such a great feature as well.
How difficult would it be to calibrate the custom prompt pack on my own data set?
- _puk
Thanks for this.
Built this myself using Gemma4, but the branching is neat. Will have to try it.
If you're packing the questions into the branches, is there a chance that the ordering of the questions can change the probability of each output?
I see you have packs, but as close to deterministic as possible is what I'm after.
- giarec
We built spinf-serve for our own needs: analyzing millions of news articles a day from
many perspectives (we run trading analysis on them), and decided to open-source it since
there seems to be interest in this kind of tool.
It serves a niche: scoring or classifying text, images and video under many questions
at once. The content is read once and every question is answered as a branch of that
read, so each extra question costs only its own tokens. Trading analysis is our use;
content moderation and inbound triage are the other obvious fits. Compared with the
alternatives we measured (vLLM with prefix caching, and dedicated decision models) it
is very token-efficient and 2 to 3 times faster on the same card; the job set and the
script are in the repo.
It runs supported open-weights models; we started with Gemma 4 and Qwen3.8. We also
tested Perplexity's pplx-decider: for hard-cut decisions (yes/no, A/B/C) a decision
model tends to be more focused, but for scoring we found that out-of-the-box models
give more sensitivity and a wider range of scores.
For our typical scoring needs we introduced prompt packs, a formalized way to ship a
set of questions for parallel scoring, with calibration and a decision rule in one file.
The server is free and open source (Apache 2.0), and it is what we run in production
for our own use cases. Let us know what you think.
There is also a hosted version with free credit if you have no GPU.