TrnRT - Open-source LLM inference engine for AWS Trainium

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

TrnRT is an open-source inference engine that brings large language models to AWS Trainium. It serves an OpenAI-compatible API from a single trn2.48xlarge instance, using NKI megakernels for efficient decode and a prefix cache for fast context reuse. With support for 1M-token context and GLM-5.3-Flash as the first model, TrnRT delivers impressive throughput: 3.57M tokens per minute at $0.040 per million tokens on spot pricing. Developers can deploy it via Docker, compile models offline, and integrate with any OpenAI client. It's a cost-effective, high-performance alternative for running LLMs on Trainium hardware.

Decode is not limited by memory bandwidth here; small instructions, kernel boundaries and exposed collectives dominate it. So each decode megakernel follows three rules: keep the working set in SBUF, use both physical cores symmetrically, and move the collectives into the kernel.
  1. coredog64

    Good on these folks for doing something useful with Trainium. I'm always surprised that the Tranium ecosystem is as poor as it is. Amazon's market cap is nearly $3T and they can't throw a couple L6s and a modest token budget at this? Making Trainium work as seamlessly as possible as an Nvidia substitute is a clear flywheel play.

  2. bookernath

    Tokens per minute is a strange metric, I usually see tok/s. A per-second figure would be more accessible in comparing this to alternatives.

  3. kevo1ution

    curious how the token/$ on trainuim compare to nvidia chips and amd. is the future "chipless" inference?

More from this day

2026-11-17