Rotary Position Embeddings (RoPE)

Notation

Notation	Description
$𝒒_{𝑚}$	$𝑚$ -th query vector without positional information
$𝒌_{𝑛}$	$𝑛$ -th key vector without positional information
$𝒒_{𝑚}^{'}$	$𝑚$ -th query vector with positional information
$𝒌_{𝑛}^{'}$	$𝑛$ -th key vector with positional information

Motivations: We Pursue Relative Positional Embedding

To incorporate positional information into the attention mechanism, we need to transform the original query and key vectors. The functions $𝑓 (\cdot, \cdot)$ encode the position indices $𝑚$ and $𝑛$ into the query and key vectors respectively, resulting in position-aware representations $𝒒_{𝑚}^{'}$ and $𝒌_{𝑛}^{'}$ .

𝒒_{𝑚}^{'} = 𝑓 (𝒒_{𝑚}, 𝑚), 𝒌_{𝑛}^{'} = 𝑓 (𝒌_{𝑛}, 𝑛)

Typically, the attention score between a query at position $𝑚$ and a key at position $𝑛$ can be represented as a function $𝑔$ that depends on both the content vectors ( $𝒒_{𝑚}$ and $𝒌_{𝑛}$ ) and their absolute positions ( $𝑚$ and $𝑛$ ) as follows:

Attn (𝒒_{𝑚}^{'}, 𝒌_{𝑛}^{'}) = 𝑔 (𝒒_{𝑚}, 𝒌_{𝑛}, 𝑚, 𝑛)

but we want the attention score to only depend on the relative position ( $𝑚 - 𝑛$ ) rather than absolute positions $𝑚$ and $𝑛$ , as relative position is easier to generalize to unseen sequence lengths.

Goal of relative positional embedding: Thus our goal is to find a function $𝑔$ which is only a function of $𝒒_{𝑚}$ , $𝒌_{𝑛}$ , and $𝑚 - 𝑛$ , instead of $𝑚$ and $𝑛$ themselves as follows:

Attn (𝒒_{𝑚}^{'}, 𝒌_{𝑛}^{'}) = 𝑔 (𝒒_{𝑚}, 𝒌_{𝑛}, 𝑚 - 𝑛)

The RoPE is a solution to this goal.

RoPE (Rotational Position Embedding)

RoPE is a positional embedding that is a function of the relative position $𝑚 - 𝑛$ , which rotates the query and key vectors and then computes the attention score, which is a function of the relative position $𝑚 - 𝑛$ .

The base idea of RoPE is to rotate the query and key vectors by a certain angle, thus their dot product is a function of the relative position $𝑚 - 𝑛$ . RoPE used the property of rotation matrix multiplication. When we need to rotate a 2-dimensional vector $(\begin{matrix} 𝑥 \\ 𝑦 \end{matrix})$ by an angle $𝜃$ , we can multiply a rotation matrix $(\begin{matrix} cos 𝜃 & - sin 𝜃 \\ sin 𝜃 & cos 𝜃 \end{matrix})$ to the vector $(\begin{matrix} 𝑥 \\ 𝑦 \end{matrix})$ , to get the rotated vector $(\begin{matrix} 𝑥 cos 𝜃 - 𝑦 sin 𝜃 \\ 𝑥 sin 𝜃 + 𝑦 cos 𝜃 \end{matrix})$ .

In the following, suppose the query and key vectors are 2-dimensional vectors. We can rotate the query and key:

𝒒_{𝑚}^{'} = 𝑓 (𝒒_{𝑚}, 𝑚) = (\begin{matrix} cos 𝑚 𝜃 & - sin 𝑚 𝜃 \\ sin 𝑚 𝜃 & cos 𝑚 𝜃 \end{matrix}) (\begin{matrix} 𝑞_{𝑚}^{1} \\ 𝑞_{𝑚}^{2} \end{matrix})

𝒌_{𝑛}^{'} = 𝑓 (𝒌_{𝑛}, 𝑛) = (\begin{matrix} cos 𝑛 𝜃 & - sin 𝑛 𝜃 \\ sin 𝑛 𝜃 & cos 𝑛 𝜃 \end{matrix}) (\begin{matrix} 𝑘_{𝑛}^{1} \\ 𝑘_{𝑛}^{2} \end{matrix})

The attention score is then:

\begin{aligned} Attn (𝒒_{𝑚}^{'}, 𝒌_{𝑛}^{'}) & = 𝒒_{𝑚}^{' 𝑇} \cdot 𝒌_{𝑛}^{'} \\ = {[(\begin{matrix} cos 𝑚 𝜃 & - sin 𝑚 𝜃 \\ sin 𝑚 𝜃 & cos 𝑚 𝜃 \end{matrix}) (\begin{matrix} 𝑞_{𝑚}^{1} \\ 𝑞_{𝑚}^{2} \end{matrix})]}^{𝑇} \cdot [(\begin{matrix} cos 𝑛 𝜃 & - sin 𝑛 𝜃 \\ sin 𝑛 𝜃 & cos 𝑛 𝜃 \end{matrix}) (\begin{matrix} 𝑘_{𝑛}^{1} \\ 𝑘_{𝑛}^{2} \end{matrix})] \\ = (\begin{matrix} 𝑞_{𝑚}^{1} & 𝑞_{𝑚}^{2} \end{matrix}) [{(\begin{matrix} cos 𝑚 𝜃 & - sin 𝑚 𝜃 \\ sin 𝑚 𝜃 & cos 𝑚 𝜃 \end{matrix})}^{𝑇} (\begin{matrix} cos 𝑛 𝜃 & - sin 𝑛 𝜃 \\ sin 𝑛 𝜃 & cos 𝑛 𝜃 \end{matrix})] (\begin{matrix} 𝑘_{𝑛}^{1} \\ 𝑘_{𝑛}^{2} \end{matrix}) \\ = (\begin{matrix} 𝑞_{𝑚}^{1} & 𝑞_{𝑚}^{2} \end{matrix}) [(\begin{matrix} cos 𝑚 𝜃 & sin 𝑚 𝜃 \\ - sin 𝑚 𝜃 & cos 𝑚 𝜃 \end{matrix}) (\begin{matrix} cos 𝑛 𝜃 & - sin 𝑛 𝜃 \\ sin 𝑛 𝜃 & cos 𝑛 𝜃 \end{matrix})] (\begin{matrix} 𝑘_{𝑛}^{1} \\ 𝑘_{𝑛}^{2} \end{matrix}) \\ = (\begin{matrix} 𝑞_{𝑚}^{1} & 𝑞_{𝑚}^{2} \end{matrix}) (\begin{matrix} cos ((𝑛 - 𝑚) 𝜃) & - sin ((𝑛 - 𝑚) 𝜃) \\ sin ((𝑛 - 𝑚) 𝜃) & cos ((𝑛 - 𝑚) 𝜃) \end{matrix}) (\begin{matrix} 𝑘_{𝑛}^{1} \\ 𝑘_{𝑛}^{2} \end{matrix}) \\ = 𝑞_{𝑚}^{1} [𝑘_{𝑛}^{1} cos ((𝑛 - 𝑚) 𝜃) - 𝑘_{𝑛}^{2} sin ((𝑛 - 𝑚) 𝜃)] \\ + 𝑞_{𝑚}^{2} [𝑘_{𝑛}^{1} sin ((𝑛 - 𝑚) 𝜃) + 𝑘_{𝑛}^{2} cos ((𝑛 - 𝑚) 𝜃)] \\ = (𝑞_{𝑚}^{1} 𝑘_{𝑛}^{1} + 𝑞_{𝑚}^{2} 𝑘_{𝑛}^{2}) cos ((𝑛 - 𝑚) 𝜃) + (𝑞_{𝑚}^{2} 𝑘_{𝑛}^{1} - 𝑞_{𝑚}^{1} 𝑘_{𝑛}^{2}) sin ((𝑛 - 𝑚) 𝜃) \\ = 𝑔 (𝒒_{𝑚}, 𝒌_{𝑛}, 𝑚 - 𝑛) \end{aligned}

thus we have shown that the attention score is a function of $𝒌_{𝑛}$ , $𝒒_{𝑚}$ , and their relative position $𝑚 - 𝑛$ , not the absolute positions $𝑚$ and $𝑛$ themselves.

RoPE Implementation (Half-and-Half Pairing)

This is the method used in your Python code and in many popular implementations like LLaMA. It is chosen for its extreme efficiency with vectorized operations. Instead of pairing adjacent dimensions (which is intuitive), this method pairs the first half of the dimensions with the second half.

For a vector $𝒒$ with dim $= 𝑑$ :

Pair 0: dimension $0$ is paired with dimension $\frac{𝑑}{2}$ . $(𝑞^{0}, 𝑞^{\frac{𝑑}{2}})$
Pair 1: dimension $1$ is paired with dimension $\frac{𝑑}{2} + 1$ . $(𝑞^{1}, 𝑞^{\frac{𝑑}{2} + 1})$
…
Pair $\frac{𝑑}{2} - 1$ : dimension $\frac{𝑑}{2} - 1$ is paired with dimension $𝑑 - 1$ . $(𝑞^{\frac{𝑑}{2} - 1}, 𝑞^{𝑑 - 1})$

In practice, the query and key vectors are not 2-dimensional vectors, but $𝑑$ -dimensional vectors ( $𝑑 % 2 = 0$ ).

𝒒_{𝑚} = (\begin{matrix} 𝑞_{𝑚}^{1} \\ 𝑞_{𝑚}^{2} \\ ⋮ \\ 𝑞_{𝑚}^{𝑑} \end{matrix}), 𝒌_{𝑛} = (\begin{matrix} 𝑘_{𝑛}^{1} \\ 𝑘_{𝑛}^{2} \\ ⋮ \\ 𝑘_{𝑛}^{𝑑} \end{matrix})

we then group the query and key vectors into $\frac{𝑑}{2}$ pairs, and rotate each pair by a different angle $𝜃_{𝑖}$ ( $𝑖 = 1, 2, \dots, \frac{𝑑}{2}$ ). Usually we make the following pairs: $[(\begin{matrix} 𝑞_{𝑚}^{0} \\ 𝑞_{𝑚}^{\frac{𝑑}{2}} \end{matrix}), (\begin{matrix} 𝑘_{𝑚}^{0} \\ 𝑘_{𝑚}^{\frac{𝑑}{2}} \end{matrix})], [(\begin{matrix} 𝑞_{𝑚}^{1} \\ 𝑞_{𝑚}^{\frac{𝑑}{2} + 1} \end{matrix}), (\begin{matrix} 𝑘_{𝑚}^{1} \\ 𝑘_{𝑚}^{\frac{𝑑}{2} + 1} \end{matrix})], \dots, [(\begin{matrix} 𝑞_{𝑚}^{\frac{𝑑}{2} - 1} \\ 𝑞_{𝑚}^{𝑑 - 1} \end{matrix}), (\begin{matrix} 𝑘_{𝑚}^{\frac{𝑑}{2} - 1} \\ 𝑘_{𝑚}^{𝑑 - 1} \end{matrix})]$ .

For each pair $𝑖$ ( $𝑖 = 0, 1, \dots, \frac{𝑑}{2} - 1$ ), we rotate by an angle $𝜃^{𝑖} = \frac{1}{𝜔^{\frac{2 𝑖}{𝑑}}} = \frac{1}{10000^{\frac{2 𝑖}{𝑑}}}$ , where $𝜔$ is a base frequency (typically $10000$ ), $𝑖$ is the $𝑖$ -th dimension $𝑖 \in [0, 1, 2, \dots, \frac{𝑑}{2} - 1]$ , and $𝑑$ is the dimension of the query and key vectors.

This means higher dimensions get rotated by larger angles, creating a spectrum of different frequencies in the positional encoding.

The rotated query vector becomes:

𝒒_{𝑚}^{'} = (\begin{matrix} 𝑞_{𝑚}^{1} cos (𝜃^{0} 𝑚) - 𝑞_{𝑚}^{\frac{𝑑}{2} + 1} sin (𝜃^{0} 𝑚) \\ 𝑞_{𝑚}^{2} cos (𝜃^{\frac{2}{𝑑}} 𝑚) - 𝑞_{𝑚}^{\frac{𝑑}{2} + 2} sin (𝜃^{\frac{2}{𝑑}} 𝑚) \\ ⋮ \\ 𝑞_{𝑚}^{\frac{𝑑}{2}} cos (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑚) - 𝑞_{𝑚}^{𝑑} sin (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑚) \\ 𝑞_{𝑚}^{1} sin (𝜃^{0} 𝑚) + 𝑞_{𝑚}^{\frac{𝑑}{2} + 1} cos (𝜃^{0} 𝑚) \\ 𝑞_{𝑚}^{2} sin (𝜃^{\frac{2}{𝑑}} 𝑚) + 𝑞_{𝑚}^{\frac{𝑑}{2} + 2} cos (𝜃^{\frac{2}{𝑑}} 𝑚) \\ ⋮ \\ 𝑞_{𝑚}^{\frac{𝑑}{2}} sin (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑚) + 𝑞_{𝑚}^{𝑑} cos (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑚) \end{matrix}) = (\begin{matrix} 𝑞_{𝑚}^{1} cos (𝜃^{0} 𝑚) - 𝑞_{𝑚}^{\frac{𝑑}{2} + 1} sin (𝜃^{0} 𝑚) \\ 𝑞_{𝑚}^{2} cos (𝜃^{\frac{2}{𝑑}} 𝑚) - 𝑞_{𝑚}^{\frac{𝑑}{2} + 2} sin (𝜃^{\frac{2}{𝑑}} 𝑚) \\ ⋮ \\ 𝑞_{𝑚}^{\frac{𝑑}{2}} cos (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑚) - 𝑞_{𝑚}^{𝑑} sin (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑚) \\ 𝑞_{𝑚}^{\frac{𝑑}{2} + 1} cos (𝜃^{0} 𝑚) + 𝑞_{𝑚}^{1} sin (𝜃^{0} 𝑚) \\ 𝑞_{𝑚}^{\frac{𝑑}{2} + 2} cos (𝜃^{\frac{2}{𝑑}} 𝑚) + 𝑞_{𝑚}^{2} sin (𝜃^{\frac{2}{𝑑}} 𝑚) \\ ⋮ \\ 𝑞_{𝑚}^{𝑑} cos (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑚) + 𝑞_{𝑚}^{\frac{𝑑}{2}} sin (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑚) \end{matrix})

Similarly for the key vector:

𝒌_{𝑛}^{'} = (\begin{matrix} 𝑘_{𝑛}^{1} cos (𝜃^{0} 𝑛) - 𝑘_{𝑛}^{\frac{𝑑}{2} + 1} sin (𝜃^{0} 𝑛) \\ 𝑘_{𝑛}^{2} cos (𝜃^{\frac{2}{𝑑}} 𝑛) - 𝑘_{𝑛}^{\frac{𝑑}{2} + 2} sin (𝜃^{\frac{2}{𝑑}} 𝑛) \\ ⋮ \\ 𝑘_{𝑛}^{\frac{𝑑}{2}} cos (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑛) - 𝑘_{𝑛}^{𝑑} sin (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑛) \\ 𝑘_{𝑛}^{1} sin (𝜃^{0} 𝑛) + 𝑘_{𝑛}^{\frac{𝑑}{2} + 1} cos (𝜃^{0} 𝑛) \\ 𝑘_{𝑛}^{2} sin (𝜃^{\frac{2}{𝑑}} 𝑛) + 𝑘_{𝑛}^{\frac{𝑑}{2} + 2} cos (𝜃^{\frac{2}{𝑑}} 𝑛) \\ ⋮ \\ 𝑘_{𝑛}^{\frac{𝑑}{2}} sin (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑛) + 𝑘_{𝑛}^{𝑑} cos (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑛) \end{matrix}) = (\begin{matrix} 𝑘_{𝑛}^{1} cos (𝜃^{0} 𝑛) - 𝑘_{𝑛}^{\frac{𝑑}{2} + 1} sin (𝜃^{0} 𝑛) \\ 𝑘_{𝑛}^{2} cos (𝜃^{\frac{2}{𝑑}} 𝑛) - 𝑘_{𝑛}^{\frac{𝑑}{2} + 2} sin (𝜃^{\frac{2}{𝑑}} 𝑛) \\ ⋮ \\ 𝑘_{𝑛}^{\frac{𝑑}{2}} cos (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑛) - 𝑘_{𝑛}^{𝑑} sin (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑛) \\ 𝑘_{𝑛}^{\frac{𝑑}{2} + 1} cos (𝜃^{0} 𝑛) + 𝑘_{𝑛}^{1} sin (𝜃^{0} 𝑛) \\ 𝑘_{𝑛}^{\frac{𝑑}{2} + 2} cos (𝜃^{\frac{2}{𝑑}} 𝑛) + 𝑘_{𝑛}^{2} sin (𝜃^{\frac{2}{𝑑}} 𝑛) \\ ⋮ \\ 𝑘_{𝑛}^{𝑑} cos (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑛) + 𝑘_{𝑛}^{\frac{𝑑}{2}} sin (𝜃^{\frac{𝑑 - 2}{𝑑}} 𝑛) \end{matrix})

When we compute the attention score between these rotated vectors, each pair contributes a term that depends on the relative position $(𝑚 - 𝑛)$ , similar to the 2D case we analyzed earlier. The different frequencies $𝜔^{2 \frac{𝑖}{𝑑}}$ allow the model to capture position-dependent patterns at different scales.

Thus we have:

$\begin{aligned} 𝒒_{𝑚}^{' 𝑇} \cdot 𝒌_{𝑛}^{'} & = [(𝑞_{𝑚}^{1} 𝑘_{𝑛}^{1} + 𝑞_{𝑚}^{\frac{𝑑}{2}} 𝑘_{𝑛}^{\frac{𝑑}{2}}) cos ((𝑛 - 𝑚) 𝜃) + (𝑞_{𝑚}^{\frac{𝑑}{2}} 𝑘_{𝑛}^{1} - 𝑞_{𝑚}^{1} 𝑘_{𝑛}^{\frac{𝑑}{2}}) sin ((𝑛 - 𝑚) 𝜃)] + \\ \dots + [(𝑞_{𝑚}^{\frac{𝑑}{2} - 1} 𝑘_{𝑛}^{\frac{𝑑}{2} - 1} + 𝑞_{𝑚}^{𝑑 - 1} 𝑘_{𝑛}^{𝑑 - 1}) cos ((𝑛 - 𝑚) 𝜃) + (𝑞_{𝑚}^{𝑑 - 1} 𝑘_{𝑛}^{\frac{𝑑}{2} - 1} - 𝑞_{𝑚}^{\frac{𝑑}{2} - 1} 𝑘_{𝑛}^{𝑑 - 1}) sin ((𝑛 - 𝑚) 𝜃)] \\ = 𝑔 (𝒒_{𝑚}, 𝒌_{𝑛}, 𝑚 - 𝑛) \end{aligned}$

Example:

Let’s walk through how the code achieves this with a concrete example where dim $= 8$ . The input query vector is $𝒒 = [𝑞^{0}, 𝑞^{1}, 𝑞^{2}, 𝑞^{3}, 𝑞^{4}, 𝑞^{5}, 𝑞^{6}, 𝑞^{7}]$ .

Pairing: The pairs will be: $(𝑞^{0}, 𝑞^{4}), (𝑞^{1}, 𝑞^{5}), (𝑞^{2}, 𝑞^{6}), (𝑞^{3}, 𝑞^{7})$ .

Frequency: There will be $\frac{𝑑}{2} = 4$ unique angles for a given position $𝑚$ : $𝜃^{0}, 𝜃^{1}, 𝜃^{2}, 𝜃^{3}$ . These angles are calculated based on the position $𝑚$ and the pair index $𝑖 \in [0, 1, 2, 3]$ :

𝜃_{𝑚}^{𝑖} = \frac{1}{10000^{\frac{2 𝑖}{𝑑}}} \cdot 𝑚

where $𝑑$ is the dimension. For example, $𝜃_{𝑚}^{0} = \frac{1}{10000^{\frac{2 * 0}{8}}} \cdot 𝑚 = \frac{1}{10000^{0}} \cdot 𝑚 = 1 \cdot 𝑚 = 𝑚$

We construct the angles for a given position $𝑚$ as $[𝜃_{𝑚}^{0}, 𝜃_{𝑚}^{1}, 𝜃_{𝑚}^{2}, 𝜃_{𝑚}^{3}, 𝜃_{𝑚}^{0}, 𝜃_{𝑚}^{1}, 𝜃_{𝑚}^{2}, 𝜃_{𝑚}^{3}]$ .

The final cos and sin tensors will therefore have this duplicated structure. For position m:

${cos}_{𝑚} = [cos (𝜃_{𝑚}^{0}), cos (𝜃_{𝑚}^{1}), cos (𝜃_{𝑚}^{2}), cos (𝜃_{𝑚}^{3}), cos (𝜃_{𝑚}^{0}), cos (𝜃_{𝑚}^{1}), cos (𝜃_{𝑚}^{2}), cos (𝜃_{𝑚}^{3})]$

${sin}_{𝑚} = [sin (𝜃_{𝑚}^{0}), sin (𝜃_{𝑚}^{1}), sin (𝜃_{𝑚}^{2}), sin (𝜃_{𝑚}^{3}), sin (𝜃_{𝑚}^{0}), sin (𝜃_{𝑚}^{1}), sin (𝜃_{𝑚}^{2}), sin (𝜃_{𝑚}^{3})]$

Now, let’s see how the rotation is applied to $𝒒 = [𝑞^{0}, 𝑞^{1}, 𝑞^{2}, 𝑞^{3}, 𝑞^{4}, 𝑞^{5}, 𝑞^{6}, 𝑞^{7}]$ .

We construct half-rotated query vector as $𝒒_{rotated} = [- 𝑞^{4}, - 𝑞^{5}, - 𝑞^{6}, - 𝑞^{7}, 𝑞^{0}, 𝑞^{1}, 𝑞^{2}, 𝑞^{3}]$ .

We then construct the query vector $𝒒_{𝑚}^{'}$ as:

$𝒒_{𝑚}^{'} = (𝒒 \cdot cos) + (𝒒_{rotated} \cdot sin)$

If we let our 2D vector be $(𝑞_{0}, 𝑞_{4})$ and the rotation angle be $𝜃_{𝑚}^{0}$ , the standard 2D rotation formulas are:

$𝑞_{0}^{'} = 𝑞_{0} cos (𝜃_{𝑚}^{0}) - 𝑞_{4} sin (𝜃_{𝑚}^{0})$
$𝑞_{4}^{'} = 𝑞_{0} sin (𝜃_{𝑚}^{0}) + 𝑞_{4} cos (𝜃_{𝑚}^{0})$

As you can see, the code perfectly implements the 2D rotation for the pair $(𝑞_{0}, 𝑞_{4})$ . This same logic applies simultaneously to all other pairs: $(𝑞_{1}, 𝑞_{5})$ , $(𝑞_{2}, 𝑞_{6})$ , $(𝑞_{3}, 𝑞_{7})$ .

$𝑞_{1}^{'} = 𝑞_{1} cos (𝜃_{𝑚}^{1}) - 𝑞_{5} sin (𝜃_{𝑚}^{1})$
$𝑞_{5}^{'} = 𝑞_{1} sin (𝜃_{𝑚}^{1}) + 𝑞_{5} cos (𝜃_{𝑚}^{1})$
$𝑞_{2}^{'} = 𝑞_{2} cos (𝜃_{𝑚}^{2}) - 𝑞_{6} sin (𝜃_{𝑚}^{2})$
$𝑞_{6}^{'} = 𝑞_{2} sin (𝜃_{𝑚}^{2}) + 𝑞_{6} cos (𝜃_{𝑚}^{2})$
$𝑞_{3}^{'} = 𝑞_{3} cos (𝜃_{𝑚}^{3}) - 𝑞_{7} sin (𝜃_{𝑚}^{3})$
$𝑞_{7}^{'} = 𝑞_{3} sin (𝜃_{𝑚}^{3}) + 𝑞_{7} cos (𝜃_{𝑚}^{3})$

Extending RoPE to Longer Contexts

Vanilla RoPE is trained with a fixed maximum context length $𝐿$ . For positions $𝑚 > 𝐿$ , the phase values $𝑚 \cdot 𝜃^{𝑖}$ exceed the range the model encountered during training, degrading performance. Three principal strategies address this: modifying the base frequency (ABF), interpolating positions uniformly (PI), and combining dimension-wise interpolation with temperature scaling (YaRN).

Recall that the phase (rotation angle) for dimension pair $𝑖$ at position $𝑚$ is:

𝜃^{𝑖} \cdot 𝑚 = \frac{𝑚}{𝜔^{\frac{2 𝑖}{𝑑}}}, 𝜔 = 10000

The goal of context extension is to keep phase values at positions $𝑚 \leq 𝐿^{'}$ within the range the model learned during training ( $𝑚 \leq 𝐿$ ). We write $𝛼 = \frac{𝐿^{'}}{𝐿}$ for the context extension ratio throughout.

We can express all extension methods in a general interpolation framework. Given the original RoPE encoding $𝑓 (𝒙_{𝑚}, 𝑚, 𝜃_{𝑖})$ , each method defines a modified encoding:

𝑓^{'} (𝒙_{𝑚}, 𝑚, 𝜃_{𝑖}) = 𝑓 (𝒙_{𝑚}, 𝑔 (𝑚), ℎ (𝜃_{𝑖}))

where $𝑔$ transforms positions and $ℎ$ transforms frequencies. The choice of $𝑔$ and $ℎ$ distinguishes the methods.

ABF (Adjusted Base Frequency)

ABF extends the context by replacing the base frequency $𝜔$ with a larger value $𝜔^{'} = 𝑠 \cdot 𝜔$ ( $𝑠 > 1$ ), directly reducing the angular frequencies and slowing phase accumulation. In the general framework, ABF sets $𝑔 (𝑚) = 𝑚$ (positions unchanged) and $ℎ (𝜃_{𝑖}) = \frac{𝜃_{𝑖}}{𝑠^{\frac{2 𝑖}{𝑑}}}$ (frequencies reduced).

Derivation

With the scaled base $𝜔^{'} = 𝑠 \cdot 𝜔$ , the phase for dimension pair $𝑖$ at position $𝑚$ becomes:

\frac{𝑚}{{(𝑠 \cdot 𝜔)}^{\frac{2 𝑖}{𝑑}}} = \frac{𝑚}{𝑠^{\frac{2 𝑖}{𝑑}} \cdot 𝜔^{\frac{2 𝑖}{𝑑}}}

Historical Context: From NTK Theory to Base Scaling

The idea of modifying the base frequency to extend context did not originate in a traditional research lab. In late June 2023 — just days after Chen et al. published Position Interpolation — a pseudonymous Reddit user, u/bloc97, posted “NTK-Aware Scaled RoPE” on the r/LocalLLaMA forum. The post demonstrated that by changing only three lines of code (replacing $𝜔 = 10000$ with a larger base), LLaMA 7B could handle 8K+ token sequences without any fine-tuning and with minimal perplexity degradation. This sparked a rapid wave of community-driven innovation: u/emozilla proposed Dynamic NTK scaling shortly after, and bloc97 followed up in July with NTK-by-parts — each refinement building on the last within weeks rather than months. By August, Meta’s Code Llama had adopted a base of $1000000$ , validating the approach at production scale. Peng et al. formalized the full method as YaRN in September 2023 (published at ICLR 2024).

The name “NTK-aware” comes from Neural Tangent Kernel theory (Jacot et al., 2018). The key theoretical connection is this: RoPE’s position encoding $[cos (𝑚 𝜃_{𝑖}), sin (𝑚 𝜃_{𝑖})]$ is essentially a random Fourier feature mapping, and NTK theory predicts that such Fourier features must preserve a spectrum of frequencies matching the target function’s complexity. When PI compresses all frequencies uniformly by $\frac{1}{𝛼}$ , it narrows the entire spectral bandwidth — destroying the high-frequency components that the network relies on for fine-grained local position discrimination. Base scaling, by contrast, distributes the interpolation pressure non-uniformly: dimensions with small $𝑖$ (high frequency, encoding local patterns) are barely changed, while dimensions with large $𝑖$ (low frequency, encoding global patterns) absorb most of the compression. This preserves the model’s sensitivity to nearby token relationships while still preventing out-of-distribution phase values at long distances.

How Modifying the Base Changes Phase Growth

We can factor the ABF phase as:

\frac{𝑚}{𝑠^{\frac{2 𝑖}{𝑑}} \cdot 𝜔^{\frac{2 𝑖}{𝑑}}} = \frac{1}{𝑠^{\frac{2 𝑖}{𝑑}}} \cdot \underset{vanilla phase}{\underset{⏟}{\frac{𝑚}{𝜔^{\frac{2 𝑖}{𝑑}}}}}

Thus the phase reduction factor compared to vanilla RoPE is $\frac{1}{𝑠^{\frac{2 𝑖}{𝑑}}}$ , which is dimension-dependent:

For $𝑖 = 0$ : reduction $= \frac{1}{𝑠^{0}} = 1$ — the highest-frequency component is unchanged.
For $𝑖 = \frac{𝑑}{2} - 1$ : reduction $= \frac{1}{𝑠^{\frac{𝑑 - 2}{𝑑}}} \approx \frac{1}{𝑠}$ — the lowest-frequency component sees the largest reduction.

ABF primarily compresses the low-frequency (high-dimensional) pairs while leaving the high-frequency (low-dimensional) pairs nearly intact, because the exponent $\frac{2 𝑖}{𝑑}$ amplifies the effect of $𝑠$ for larger $𝑖$ .

Figure 1. ABF phase reduction factor

\frac{1}{𝑠^{\frac{2 𝑖}{𝑑}}}

across dimension pairs (

𝑑 = 128

). Larger base scaling factor

𝑠

produces stronger compression, concentrated in higher-dimensional (low-frequency) pairs. All curves start at

1

for

𝑖 = 0

Choosing $𝑠$ : NTK-Aware Scaling

To extend from context length $𝐿$ to $𝐿^{'}$ , we require the lowest-frequency pair ( $𝑖 = \frac{𝑑}{2} - 1$ ) at position $𝐿^{'}$ to not exceed its vanilla phase at position $𝐿$ :

\frac{𝐿^{'}}{𝑠^{\frac{𝑑 - 2}{𝑑}} \cdot 𝜔^{\frac{𝑑 - 2}{𝑑}}} \leq \frac{𝐿}{𝜔^{\frac{𝑑 - 2}{𝑑}}}

𝑠^{\frac{𝑑 - 2}{𝑑}} \geq \frac{𝐿^{'}}{𝐿}

𝑠 \geq {(\frac{𝐿^{'}}{𝐿})}^{\frac{𝑑}{𝑑 - 2}}

The minimum value gives the NTK-aware base:

𝜔^{'} = 𝜔 \cdot {(\frac{𝐿^{'}}{𝐿})}^{\frac{𝑑}{𝑑 - 2}}

Practical Base Frequency Values

The theoretical analysis above predicts the minimum base frequency needed for a target context length. In practice, several prominent models adopt large base values that align with these bounds:

Code Llama: $𝜔 = 1000000$ (extending from 4K to 100K tokens).
Meta long-context models: $𝜔 = 500000$ (extending LLaMA-family models to 32K–128K tokens).

Men et al. (2024) establish a tighter theoretical bound by requiring the cumulative rotation $𝐵_{𝑚, 𝜃} = \sum_{𝑖} cos (𝑚 𝜃_{𝑖}) \geq 0$ for effective positional discrimination. Their analysis yields minimum base requirements that grow rapidly with context length: approximately $4.3 \times 10^{3}$ for 4K context, $6.4 \times 10^{5}$ for 32K, and $7.8 \times 10^{6}$ for 128K. The values used in practice ( $500000$ – $1000000$ ) sit comfortably above these theoretical minima.

A key advantage of ABF over pure position interpolation is that it preserves local positional resolution — nearby tokens at low dimension pairs still see approximately the same phase differences as in vanilla RoPE, since the high-frequency components are nearly unchanged. The cost is that inter-dimensional phase ratios are distorted, as we will see when comparing with PI and YaRN below.

Limitations of Large Scaling Factors

While ABF successfully extends context length, increasing the scaling factor $𝑠$ introduces growing distortions that limit its effectiveness. Recall from the stability analysis that ABF modifies the inter-dimensional phase ratio by an extra factor of $𝑠^{\frac{2 (𝑗 - 𝑖)}{𝑑}}$ . As $𝑠$ grows large, this factor becomes substantial for dimension pairs that are far apart in index — meaning the model encounters inter-dimensional relationships that deviate significantly from what it learned during pre-training. In the extreme, with $𝑠 = 100$ and $𝑑 = 128$ , the ratio between the lowest and highest frequency pairs is distorted by a factor exceeding $𝑠^{\frac{𝑑 - 2}{𝑑}} \approx 97 \times$ , a regime well outside the model’s training distribution.

Research by Liu et al. (2023) on scaling laws for RoPE-based extrapolation reveals a counterintuitive finding: the default base of $𝜔 = 10000$ actually yields the worst extrapolation performance when models are fine-tuned for longer contexts. Performance improves when the base is adjusted in either direction. Smaller bases (e.g., $𝜔 = 500$ ) produce smooth, gradual perplexity degradation that enables nearly unlimited extrapolation potential, while larger bases (e.g., $𝜔 = 1000000$ ) produce excellent performance within a well-defined range but exhibit a sharp perplexity cliff beyond it. This non-monotonic behavior arises from a critical dimension $𝑑_{extra}$ — the dimension index beyond which wavelengths are shorter than the training context $𝐿$ . For LLaMA 2 7B with $𝑑 = 128$ , this critical dimension is approximately $𝑑_{extra} = 92$ . Dimensions beyond $𝑑_{extra}$ have already completed many rotation cycles during training and can extrapolate safely, while dimensions below it have not seen a full cycle and require interpolation. The interplay between these two regimes explains why the default base sits at a local pessimum: it places too many dimensions in the ambiguous intermediate zone.

Men et al. (2024) reinforce this picture with their finding of “superficial long-context ability.” Models with an insufficiently large base can achieve low perplexity on long sequences — appearing to handle extended contexts — while actually failing at tasks that require precise long-range retrieval. The perplexity metric masks the model’s inability to attend to distant tokens with high specificity, because the cumulative rotation bound $𝐵_{𝑚, 𝜃} = \sum_{𝑖} cos (𝑚 𝜃_{𝑖})$ approaches zero before the perplexity fully degrades. This gap between perplexity and retrieval accuracy is an important caveat when evaluating any base-scaling approach.

PI (Position Interpolation)

Position Interpolation takes the simplest possible approach: instead of modifying frequencies, it scales down all positions to fit within the trained range. In the general framework, PI sets $𝑔 (𝑚) = \frac{𝑚}{𝛼}$ (positions compressed) and $ℎ (𝜃_{𝑖}) = 𝜃_{𝑖}$ (frequencies unchanged).

Derivation

Given the context extension ratio $𝛼 = \frac{𝐿^{'}}{𝐿}$ , PI maps each position:

𝑚 \to \frac{𝑚}{𝛼}

The frequencies $𝜃^{𝑖}$ remain unchanged. The phase at position $𝑚$ becomes:

(\frac{𝑚}{𝛼}) \cdot 𝜃^{𝑖} = \frac{𝑚}{𝛼 \cdot 𝜔^{\frac{2 𝑖}{𝑑}}}

How Position Interpolation Changes Phase

We can factor the PI phase as:

\frac{𝑚}{𝛼 \cdot 𝜔^{\frac{2 𝑖}{𝑑}}} = \frac{1}{𝛼} \cdot \underset{vanilla phase}{\underset{⏟}{\frac{𝑚}{𝜔^{\frac{2 𝑖}{𝑑}}}}}

The reduction factor is $\frac{1}{𝛼}$ , which is uniform across all dimensions.

At the extended boundary $𝑚 = 𝐿^{'}$ :

(\frac{𝐿^{'}}{𝛼}) \cdot 𝜃^{𝑖} = 𝐿 \cdot 𝜃^{𝑖}

The phase at position $𝐿^{'}$ under PI equals the phase at position $𝐿$ under vanilla RoPE for every dimension $𝑖$ . The model sees the same range of phase values it was trained on, just at a finer positional granularity.

Stability of Phase Ratios

Why PI preserves inter-dimensional relationships. Under vanilla RoPE, the inter-dimensional phase ratio between pairs $𝑖$ and $𝑗$ is:

\frac{𝜃^{𝑖} \cdot 𝑚}{𝜃^{𝑗} \cdot 𝑚} = 𝜔^{\frac{2 (𝑗 - 𝑖)}{𝑑}}

Under PI, this ratio is preserved exactly (the $\frac{1}{𝛼}$ cancels):

\frac{(\frac{𝑚}{𝛼}) \cdot 𝜃^{𝑖}}{(\frac{𝑚}{𝛼}) \cdot 𝜃^{𝑗}} = \frac{𝜃^{𝑖}}{𝜃^{𝑗}} = 𝜔^{\frac{2 (𝑗 - 𝑖)}{𝑑}}

Under ABF, the modified frequencies $\frac{1}{{(𝑠 𝜔)}^{\frac{2 𝑖}{𝑑}}}$ change this ratio to:

{(𝑠 𝜔)}^{\frac{2 (𝑗 - 𝑖)}{𝑑}} = 𝑠^{\frac{2 (𝑗 - 𝑖)}{𝑑}} \cdot 𝜔^{\frac{2 (𝑗 - 𝑖)}{𝑑}}

The extra factor $𝑠^{\frac{2 (𝑗 - 𝑖)}{𝑑}}$ distorts the learned inter-dimensional relationships. Since PI preserves these ratios exactly, it typically requires less fine-tuning and is more stable when extending context length.

Limitations of Uniform Interpolation

While PI elegantly preserves inter-dimensional phase ratios, the uniform $\frac{1}{𝛼}$ compression has a significant drawback. High-frequency dimension pairs (small $𝑖$ ) already have short wavelengths — they encode fine-grained local position differences. Compressing these by $\frac{1}{𝛼}$ reduces the model’s ability to distinguish nearby positions, effectively losing high-frequency positional information.

Low-frequency pairs (large $𝑖$ ), on the other hand, have wavelengths much longer than the training context $𝐿$ . For these dimensions, the model has never seen a full rotation cycle during training, so the phase values encountered at positions up to $𝐿$ occupy only a small arc. These dimensions genuinely need interpolation to avoid out-of-distribution phase values at positions beyond $𝐿$ . But high-frequency dimensions already complete many full rotations within $[0, 𝐿]$ , so they can tolerate positions beyond $𝐿$ without extrapolation problems.

This observation — that different dimensions require different amounts of interpolation — motivates the more sophisticated YaRN method.

YaRN (Yet Another RoPE ExtensioN)

YaRN addresses the limitations of both ABF and PI by combining dimension-wise interpolation (NTK-by-parts) with attention temperature scaling. Rather than applying a single strategy uniformly, YaRN classifies each dimension pair according to how much interpolation it needs and blends PI with identity scaling accordingly.

Wavelength Analysis

Each dimension pair $𝑖$ has a characteristic wavelength — the number of positions for a full $2 𝜋$ rotation:

𝜆_{𝑖} = 2 \frac{𝜋}{𝜃^{𝑖}} = 2 𝜋 \cdot 𝜔^{\frac{2 𝑖}{𝑑}}

We define the ratio of the training context length to this wavelength:

𝑟 (𝑖) = \frac{𝐿}{𝜆_{𝑖}} = \frac{𝐿}{2 𝜋 \cdot 𝜔^{\frac{2 𝑖}{𝑑}}}

This ratio $𝑟 (𝑖)$ tells us how many full rotation cycles dimension $𝑖$ completes within the training window:

High ratio ( $𝑟 (𝑖) ≫ 1$ ): the model has seen many full cycles during training. This dimension encodes local positional patterns and can tolerate extrapolation to positions beyond $𝐿$ without problems. No interpolation is needed.
Low ratio ( $𝑟 (𝑖) ≪ 1$ ): the model has seen only a small arc of the full rotation. Positions beyond $𝐿$ would produce phase values the model has never encountered. Full interpolation (as in PI) is needed.
Intermediate ratio: a blend of interpolation and identity is appropriate.

NTK-by-Parts Interpolation

YaRN uses a ramp function $𝛾 (𝑟)$ to smoothly transition between full interpolation and no interpolation, based on the ratio $𝑟 (𝑖)$ :

𝛾 (𝑟) = {\begin{cases} 0 & if 𝑟 < 𝛼_{lo} \\ 1 & if 𝑟 > 𝛼_{hi} \\ \frac{𝑟 - 𝛼_{lo}}{𝛼_{hi} - 𝛼_{lo}} & otherwise \end{cases}

where $𝛼_{lo}$ and $𝛼_{hi}$ are hyperparameters controlling the transition boundaries. For the LLaMA model family, the recommended values are $𝛼_{lo} = 1$ and $𝛼_{hi} = 32$ .

The choice of these boundaries has a precise physical interpretation tied to rotation cycles. A dimension pair with $𝑟 (𝑖) < 1$ — that is, $𝑟 (𝑖) < 𝛼_{lo}$ — has a wavelength longer than the training context $𝐿$ . The model has seen less than one full rotation cycle for this dimension during pre-training, so any position beyond $𝐿$ pushes the phase into entirely uncharted territory. These dimensions are the most vulnerable to extrapolation failure and must receive full PI-style interpolation. At the other extreme, $𝑟 (𝑖) > 32$ means the model has observed 32 or more complete rotation cycles during training. The positional patterns at these frequencies are thoroughly learned; the model can reliably extrapolate because the periodic structure is well established. Leaving these dimensions unchanged preserves the fine-grained local position sensitivity that PI would destroy.

This connects directly to the critical dimension concept from Liu et al. (2023): the boundary $𝑟 (𝑖) = 1$ corresponds roughly to the index $𝑑_{extra}$ where dimensions transition from having seen partial rotations to having seen full cycles. YaRN’s ramp function effectively operationalizes this theoretical boundary, with $𝛼_{lo} = 1$ marking the onset of safe extrapolation and $𝛼_{hi} = 32$ marking the point of full confidence. The linear transition between them provides a smooth blend that avoids abrupt changes in the frequency spectrum — a design choice validated empirically across the LLaMA model family.

The modified frequency for dimension pair $𝑖$ is then:

ℎ (𝜃_{𝑖}) = (1 - 𝛾 (𝑟 (𝑖))) \cdot \frac{𝜃_{𝑖}}{𝛼} + 𝛾 (𝑟 (𝑖)) \cdot 𝜃_{𝑖}

We can factor this to understand the per-dimension behavior:

ℎ (𝜃_{𝑖}) = \underset{vanilla freq.}{\underset{⏟}{𝜃_{𝑖}}} \cdot [𝛾 (𝑟 (𝑖)) + \frac{1 - 𝛾 (𝑟 (𝑖))}{𝛼}]

The three regimes are:

When $𝛾 = 0$ (low-frequency dimensions, $𝑟 (𝑖) < 𝛼_{lo}$ ): $ℎ (𝜃_{𝑖}) = \frac{𝜃_{𝑖}}{𝛼}$ , identical to PI — full interpolation.
When $𝛾 = 1$ (high-frequency dimensions, $𝑟 (𝑖) > 𝛼_{hi}$ ): $ℎ (𝜃_{𝑖}) = 𝜃_{𝑖}$ , the frequency is unchanged — no interpolation at all.
When $0 < 𝛾 < 1$ (transition region): $ℎ (𝜃_{𝑖}) = 𝜃_{𝑖} \cdot [𝛾 + \frac{1 - 𝛾}{𝛼}]$ , a linear blend between PI and identity.

Figure 2. YaRN NTK-by-parts interpolation with

𝛼 = 4

𝐿 = 4096

𝜔 = 10000

𝑑 = 128

𝛼_{lo} = 1

𝛼_{hi} = 32

. The ramp function

𝛾

(blue) transitions from

0

1

across dimension pairs. The effective frequency scaling factor (red) interpolates between

\frac{1}{𝛼} = 0.25

(full PI, dotted gray) and

1

(no change, dashed gray). Low-frequency dimensions (small

𝑖

) receive full interpolation; high-frequency dimensions (large

𝑖

) are left unchanged.

Temperature Scaling

NTK-by-parts interpolation solves the dimension-wise frequency problem, but it introduces a subtler issue: the modified frequencies change the entropy of the attention distribution. When we interpolate some dimensions but not others, the effective magnitude of the dot-product attention scores shifts, causing the softmax to become either too sharp or too diffuse compared to the original model.

YaRN corrects this with a temperature parameter $𝑡$ applied to the attention logits:

Attn (𝒒_{𝑚}^{'}, 𝒌_{𝑛}^{'}) = softmax (\frac{𝒒_{𝑚}^{' 𝑇} 𝒌_{𝑛}^{'}}{𝑡 \cdot \sqrt{𝑑}})

The temperature is derived from the extension ratio $𝛼$ via an empirically calibrated formula:

\sqrt{\frac{1}{𝑡}} = 0.1 \cdot ln (𝛼) + 1

Solving for $𝑡$ :

𝑡 = \frac{1}{{(0.1 \cdot ln (𝛼) + 1)}^{2}}

Since $𝛼 > 1$ , we have $0.1 \cdot ln (𝛼) + 1 > 1$ , which gives $𝑡 < 1$ . Dividing by $𝑡 < 1$ scales up the attention logits, sharpening the attention distribution. This counteracts the entropy increase caused by the modified frequencies, restoring the attention pattern to a distribution similar to what the model learned during training.

For example, with $𝛼 = 4$ : $\sqrt{\frac{1}{𝑡}} = 0.1 \cdot ln (4) + 1 \approx 1.139$ , so $𝑡 \approx 0.771$ .

The formula $\sqrt{\frac{1}{𝑡}} = 0.1 \cdot ln (𝛼) + 1$ was found by empirical fitting, not theoretical derivation. Peng et al. applied NTK-by-parts with various scaling factors $𝛼$ to LLaMA 7B, 13B, 33B, and 65B models without fine-tuning, sweeping over temperature values $𝑡$ for each configuration to find the one minimizing perplexity. When they plotted $\sqrt{\frac{1}{𝑡}}$ against $ln (𝛼)$ , the relationship turned out to be remarkably linear — and consistent across all four model sizes. They validated this across 896 documents from RedPajama at different token positions, confirming that the optimal temperature depends primarily on the extension ratio rather than on model size or input content.

To understand why NTK-by-parts increases entropy in the first place, consider what happens to the dot-product attention scores. Interpolated dimensions (those with $𝛾 \approx 0$ ) have their frequencies reduced by $\frac{1}{𝛼}$ , which compresses the phase differences between tokens — positions that were once far apart in rotated space now appear closer. These dimensions contribute smaller dot products to the attention score. Meanwhile, unchanged dimensions (those with $𝛾 \approx 1$ ) contribute normal-magnitude dot products. The net effect is that the range of attention scores across different key positions shrinks: the gap between the most-attended and least-attended tokens narrows. When softmax operates on these compressed scores, it produces a flatter, more uniform distribution — higher entropy. The model attends to everything more equally, losing its ability to focus on the most relevant tokens. Dividing by $𝑡 < 1$ is equivalent to multiplying the attention logits by $\frac{1}{𝑡} > 1$ , which stretches the score range back out, restores the sharpness of the attention distribution, and brings the entropy profile closer to what the model learned during pre-training.

We note that the temperature formula is architecture-dependent. The coefficient $0.1$ in $\sqrt{\frac{1}{𝑡}} = 0.1 \cdot ln (𝛼) + 1$ was fit specifically for LLaMA models. For Mistral-7B, the optimal coefficient shifts to approximately $0.07$ , reflecting differences in head dimension, number of attention heads, and the learned attention patterns. In the general parametric form $\sqrt{\frac{1}{𝑡}} = 𝑎 \cdot ln (𝛼) + 𝑏$ , both $𝑎$ and $𝑏$ may need re-fitting when applying YaRN to a new architecture — though the logarithmic relationship with $𝛼$ appears to be universal.

Dynamic NTK

In practice, the extension ratio need not be fixed at inference time. Dynamic NTK computes the scaling factor on the fly based on the current sequence length $𝑙^{'}$ :

𝛼 = max (\frac{𝑙^{'}}{𝐿}, 1)

When $𝑙^{'} \leq 𝐿$ , we have $𝛼 = 1$ and YaRN reduces to vanilla RoPE — no modification is applied. As $𝑙^{'}$ exceeds $𝐿$ , the interpolation gradually activates. This allows a single model to handle both short and long sequences without committing to a fixed extension ratio at deployment time.

Empirical Performance

We can now examine how YaRN performs in practice relative to other extension methods. The following table summarizes perplexity results on the Proof-pile evaluation set using a sliding window of $𝑆 = 256$ tokens, drawn from the YaRN paper (Peng et al., 2023):

Model	Method	Perplexity (65K)	Perplexity (128K)
LLaMA 2 7B	YaRN ( $𝑠 = 16$ )	2.42	—
LLaMA 2 7B	YaRN ( $𝑠 = 32$ )	$\sim 2.37$	2.37
LLaMA 2 13B	YaRN ( $𝑠 = 32$ )	—	2.24
Code Llama 7B	ABF ( $𝜔 = 10^{6}$ )	2.55	2.54
Together AI 7B	PI	$> 32$ K blow-up	—

YaRN achieves the lowest perplexity at 128K tokens with the 13B model, outperforming Code Llama despite the latter using a much larger base frequency of $𝜔 = 1000000$ . PI, while theoretically elegant, fails catastrophically beyond its trained extension range — perplexity explodes past 32K tokens for the Together AI model. On passkey retrieval — a stricter test of whether the model can actually attend to a specific token at arbitrary distances — YaRN ( $𝑠 = 32$ , 128K) achieves 99.4% accuracy, matching Code Llama’s 100K result.

The training efficiency comparison is equally striking. NTK-aware scaling from Xiong et al. requires approximately 64,000 A100 GPU-hours of continued pre-training. PI (Chen et al.) brings this down to roughly 640 GPU-hours. YaRN requires only 384 GPU-hours total — 256 for the initial 64K extension plus 128 for the additional 128K stage — using approximately 0.1% of the original pre-training data and converging in just 400 optimization steps for the first stage. This makes YaRN the most compute-efficient method by a significant margin while achieving superior or comparable results.

An important caveat emerges from these results: perplexity alone is not a reliable indicator of long-context capability. YaRN with $𝑠 = 32$ achieves similar perplexity to $𝑠 = 16$ , yet the $𝑠 = 32$ model scores meaningfully higher on passkey retrieval, demonstrating better ability to attend to distant tokens. This gap suggests that perplexity primarily captures local language modeling quality, while retrieval tasks probe whether the model can genuinely leverage the full extended context. When evaluating RoPE extension methods, we recommend testing with retrieval-based benchmarks (such as passkey retrieval or needle-in-a-haystack) alongside perplexity.

From a practical standpoint, YaRN incurs zero computational overhead at inference time. The modified frequencies are pre-computed and cached, just as in vanilla RoPE, and the temperature scaling amounts to a single scalar multiplication on the attention logits. YaRN is natively supported in HuggingFace Transformers via rope_type: "yarn", and has been adopted by production models including Qwen, DeepSeek, and later LLaMA variants — making it the de facto standard for RoPE-based context extension.

Comparison of Extension Methods

Method	Phase at position $𝑚$	Reduction factor	Scaling type
Vanilla RoPE	$\frac{𝑚}{𝜔^{\frac{2 𝑖}{𝑑}}}$	—	—
Base scaling ( $𝜔 \to 𝛼 𝜔$ )	$\frac{𝑚}{{(𝛼 𝜔)}^{\frac{2 𝑖}{𝑑}}}$	$\frac{1}{𝛼^{\frac{2 𝑖}{𝑑}}}$	Dimension-dependent
ABF ( $𝜔 \to 𝜔 𝛼^{\frac{𝑑}{𝑑 - 2}}$ )	$\frac{𝑚}{𝜔^{\frac{2 𝑖}{𝑑}} \cdot 𝛼^{\frac{2 𝑖}{𝑑 - 2}}}$	$\frac{1}{𝛼^{\frac{2 𝑖}{𝑑 - 2}}}$	Dim.-dependent (calibrated)
PI ( $𝑚 \to \frac{𝑚}{𝛼}$ )	$\frac{𝑚}{𝛼 \cdot 𝜔^{\frac{2 𝑖}{𝑑}}}$	$\frac{1}{𝛼}$	Uniform
YaRN (NTK-by-parts)	$𝑚 \cdot ℎ (𝜃_{𝑖})$ , with $ℎ$ per-dim.	$𝛾 + \frac{1 - 𝛾}{𝛼}$	Dimension-adaptive + temp.

where $𝛼 = \frac{𝐿^{'}}{𝐿}$ is the context extension ratio and $𝛾 = 𝛾 (𝑟 (𝑖))$ is the ramp function evaluated at each dimension.

We can summarize the key trade-offs as follows:

Base scaling and ABF modify the frequency spectrum in a dimension-dependent way, preserving high-frequency (local) positional resolution at the cost of distorting inter-dimensional phase ratios.
PI applies a uniform reduction that perfectly preserves phase ratios but sacrifices high-frequency resolution.
YaRN achieves the best of both worlds: it preserves high-frequency dimensions (where the model can extrapolate) while interpolating only the low-frequency dimensions (where extrapolation would fail), and corrects the resulting entropy shift via temperature scaling.

Figure 3. Comparison of phase reduction factors with

𝛼 = 4

𝑑 = 128

. Vanilla RoPE (dashed gray) has no reduction. PI (green) applies a uniform

\frac{1}{4}

reduction. Base scaling (blue) and ABF (red) reduce higher dimensions more aggressively. YaRN (orange) transitions smoothly from

\frac{1}{𝛼}

at low-frequency dimensions to

1

at high-frequency dimensions, combining the strengths of PI and ABF.

Notation

Motivations: We Pursue Relative Positional Embedding

RoPE (Rotational Position Embedding)

RoPE Implementation (Half-and-Half Pairing)

Example:

Extending RoPE to Longer Contexts

ABF (Adjusted Base Frequency)

Derivation

Historical Context: From NTK Theory to Base Scaling

How Modifying the Base Changes Phase Growth

Choosing 𝑠: NTK-Aware Scaling

Practical Base Frequency Values

Limitations of Large Scaling Factors

PI (Position Interpolation)

Derivation

How Position Interpolation Changes Phase

Stability of Phase Ratios

Limitations of Uniform Interpolation

YaRN (Yet Another RoPE ExtensioN)

Wavelength Analysis

NTK-by-Parts Interpolation

Temperature Scaling

Dynamic NTK

Empirical Performance

Comparison of Extension Methods

Choosing $𝑠$ : NTK-Aware Scaling