TrnRT - Open-source LLM inference engine for AWS Trainium
Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium
TrnRT is an open-source inference engine that brings large language models to AWS Trainium. It serves an OpenAI-compatible API from a single trn2.48xlarge instance, using NKI megakernels for efficient decode and a prefix cache for fast context reuse. With support for 1M-token context and GLM-5.3-Flash as the first model, TrnRT delivers impressive throughput: 3.57M tokens per minute at $0.040 per million tokens on spot pricing. Developers can deploy it via Docker, compile models offline, and integrate with any OpenAI client. It's a cost-effective, high-performance alternative for running LLMs on Trainium hardware.
Decode is not limited by memory bandwidth here; small instructions, kernel boundaries and exposed collectives dominate it. So each decode megakernel follows three rules: keep the working set in SBUF, use both physical cores symmetrically, and move the collectives into the kernel.
- coredog64
Good on these folks for doing something useful with Trainium. I'm always surprised that the Tranium ecosystem is as poor as it is. Amazon's market cap is nearly $3T and they can't throw a couple L6s and a modest token budget at this? Making Trainium work as seamlessly as possible as an Nvidia substitute is a clear flywheel play.
- bookernath
Tokens per minute is a strange metric, I usually see tok/s. A per-second figure would be more accessible in comparing this to alternatives.
- kevo1ution
curious how the token/$ on trainuim compare to nvidia chips and amd. is the future "chipless" inference?