MIT and Adobe add a second stream of memory that cuts language model perplexity

Latent Lattice Language Models

Instead of deepening layers or adding latent reasoning steps, MIT and Adobe researchers give each layer its own horizontal latent stream that persists across tokens, running alongside the vertical residual stream. On 20B FineWeb-Edu tokens, a 406M-parameter Lattice model drops validation perplexity from 15.41 to 14.40 versus a parameter-matched Transformer, and combining it with hyper-connections and input residuals reaches 13.61. Gains hold across Gated DeltaNet, Mamba-2, and hybrid backbones, pointing to persistent per-layer state as a separate axis for scaling computation.

These results identify persistent per-layer state as a complementary axis for scaling language-model computation.

More from this day

2026-11-17