Attention Heads Are Independent and Additive: A Mathematical Framework for Reverse-Engineering Transformers
A Mathematical Framework for Transformer Circuits (2021)
This paper introduces a mathematical framework for mechanistic interpretability of transformer language models, focusing on attention-only models with up to two layers. The authors show that zero-layer transformers model bigram statistics, one-layer transformers are ensembles of bigram and skip-trigram models, and two-layer transformers can implement complex algorithms via composition of attention heads, notably induction heads that explain in-context learning. Key insights include the residual stream as a communication channel, attention heads as independent additive operations, and decomposition into QK and OV circuits. The framework provides a foundation for reverse-engineering larger models.
We find that specific attention heads that we term “induction heads” can explain in-context learning in these small models, and that these heads only develop in models with at least two attention layers.
- myself248
Nothing about B-H curves or magnetization current, must be that other kind of transformer, in that other kind of circuit.
- _jayhack_
Shocking to me how little interest the general public has in mechinterp given the alien capabilities demonstrated by LLMs - this and the subsequent transformer-circuits.pub publications will be seen as classic, foundational work in a few years
- amelius
> A previous project, the Distill Circuits thread, has attempted to reverse engineer vision models, but so far there hasn’t been a comparable project for transformers or language models.
How successful was that?