Researchers Build a Transformer Rival That Drops Attention Entirely
Attention Is Not What You Need
A new paper asks whether explicit self-attention is really necessary for strong performance. The authors reframe multi-head attention as tensor lifting and propose a Causal Grassmann layer that encodes token pairs as 2D subspaces via Plücker coordinates, then fuses them back through gated mixing. On Wikitext-2, 13–18M-parameter Grassmann models land within 10–15% of size-matched Transformers; on SNLI, a Grassmann-Plücker head on DistilBERT slightly edges out a Transformer head.
This mechanism is extremely expressive but mathematically opaque, because after many layers it becomes very hard to describe the model with a small family of explicit invariants.