Why LLMs Need Positional Embeddings—and How RoPE Changed the Game

Understanding Positional Embeddings in Modern LLMs

Why LLMs Need Positional Embeddings—and How RoPE Changed the Game

Sebastian Raschka walks through how large language models encode token order, from GPT-2's learned absolute position vectors to the sinusoidal encodings of the original transformer and the rotary position embeddings (RoPE) used in most modern open-weight models. With visual examples, a little math, and a small PyTorch implementation, he explains why explicit position signals still matter even when causal attention masks provide implicit order, and surveys which positional choices appear in 2026 architectures.

Interestingly, this causal mask gives the computation some implicit order information. In our example, “jumps” can attend to “red” in the first sentence and “gray” in the second. So, a causal LLM can theoretically already distinguish these sentences even without explicit positional embeddings.
  1. programjames

    This article is only about understanding the implementation of positional embeddings, but to really understand positional embeddings you have to understand the theory. Positional embeddings are group representations. This is how you find them, and this is also how you parameterize them to make them differentiable/learnable. To give a few examples:

    1. Rotary positional embeddings are representations of the translation group. The n-dimensional translation group ℝⁿ is abelian, so its irreducible representations are all one-dimensional ρ: ℝⁿ → ℂ, specified by a momentum vector m: ρₘ(x) = exp(i⟨m, x⟩). You can get a multi-dimensional representation by choosing several m, which are your learnable parameters.

    2. If you wanted RoPE embeddings to also preserve rotation (perhaps you have scenes viewed at arbitrary angles), you are looking for representations of the special Euclidean group. The irreducible representations in four dimensions are your familiar particles: electrons, protons, etc., and in general are parameterized by a momentum vector and a discrete spin vector. You would have to do some kind of annealing process to train the spin vector (or you could just make it constant).

    3. If you want to do attention on quadtrees, well, what is the group? Suppose every node in the quadtree has local coordinates scaled to [0, 1]. Moving up/down/to a neighbor in the quadtree requires an affine transformation of the local coordinates, which compose, so we can label every node by its affin […]

More from this day

2026-11-15