Hetzner Tests LLM Inference with Qwen on Its Own Infrastructure
Hetzner is working on LLM Inference
I tried Hetzner's new experimental LLM inference service, which offers a free, OpenAI-compatible API running on their own hardware. While currently limited to a single Qwen model with no SLA, the performance was surprisingly fast. This move suggests Hetzner might leverage its efficient data centers to turn spare GPU capacity into a low-margin inference product, though the lack of massive multi-GPU clusters remains a question.
If Hetzner keeps serving one or two smaller models, I do not really see it becoming an important inference provider. That would be a cool experiment, but not much more.
- swiftcoder
It would certainly be interesting to have a highly respected EU-native inference provider, if only to make the regulatory gods happy
- pmg1991
Very soon we will be having reseller programs for inference, this will be just like web hosting reseller. After big players, small players will also start entering in this field.
I'm waiting for that day so that inference will be affordable just like web hosting. 200$ per month is in no way affordable by everyone.
- ano-ther
Good to see more developments in this space. I quite like this service, which is a little further than Hetzner and has several models to choose from: https://www.infomaniak.com/en/hosting/ai-services
- mark_l_watson
This seems like a smart move, given their ability to host efficiently. I approve of efforts to make the cost of inference for smaller useful models slowly approach 'close to zero' and there are many good paths for getting there. It is useful for companies to get fast hosting for the class of smaller models they may end up hosting in house.
- NetOpWibby
This is interesting because I thought Hetzner was anti-crypto? LLMs aren't the same but they're often lumped in with crypto as "things no one wants."