Emulating ALiBi with RoPE
LLMs use positional encodings like RoPE and ALiBi to track token positions. This post shows how to emulate ALiBi's linear bias using a fixed RoPE pair, by adding two dimensions to Q and K and tuning the rotation frequency. The author proves that ALiBi is the small-angle limit of this RoPE pair, and validates the approach by reproducing BLOOM-560M's results. Practical limitations, such as choosing frequencies from an existing RoPE base, are also discussed.
So ALiBi is equivalent to the small-angle limit of a fixed RoPE pair, so as long as you make θ small enough for long contexts, which in turn causes N to be large.