Encoders vs. Decoders: The Latency Trade-off Nobody Talks About
Strands Decider: Why Not an Encoder?
After launching strands-decider-2B, a modified LLM, people asked why not use an encoder like BERT. I compared five designs: a Qwen3.5-based decoder, encoder-decoder models, and encoder-only models. Encoders won on latency (35ms vs 65ms) with similar accuracy, but scaled worse with longer prompts. The results are confounded, but neither approach is a clear winner at this size.
With half the latency and similar performance, this is looking like a big win for e1b. But things get a little more complicated when we dive into the data.