Group Relative Policy Optimization (GRPO)

Objective Function

GRPO [1] maximizes the expected total reward of a group of responses. Given a question $𝑞$ sampled from dataset $𝒟︀$ and a group of $𝐺$ responses ${𝑜_{𝑖}}_{𝑖 = 1}^{𝐺}$ sampled from the old policy $𝜋_{𝜃_{old}}$ , the objective is:

\begin{aligned} 𝒥︀ (𝜃) & = 𝔼_{{(𝑞, 𝑎) \sim 𝒟︀, {𝑜_{𝑖}}_{𝑖 = 1}^{𝐺} \sim 𝜋_{𝜃_{old}} (\cdot | 𝑞)}} \\ [\frac{1}{𝐺} \sum_{𝑖 = 1}^{𝐺} \frac{1}{| 𝑜_{𝑖} |} \sum_{𝑡 = 1}^{| 𝑜_{𝑖} |} [min (𝑟_{𝑖, 𝑡} (𝜃) {\hat{𝐴}}_{𝑖, 𝑡}, clip (𝑟_{𝑖, 𝑡} (𝜃), 1 - 𝜀, 1 + 𝜀) {\hat{𝐴}}_{𝑖, 𝑡})]] \end{aligned}

where:

$(𝑞, 𝑎) \sim 𝒟︀$ is the question-answer pair from dataset $𝒟︀$
$𝐺$ is the group size; $𝑜_{𝑖}$ is the $𝑖$ -th response; $| 𝑜_{𝑖} |$ is the length of the $𝑖$ -th response
$𝑟_{𝑖, 𝑡} (𝜃) = \frac{𝜋_{𝜃} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡})}{𝜋_{𝜃_{old}} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡})}$ is the probability ratio of the $𝑡$ -th token in the $𝑖$ -th response
$𝜋_{𝜃} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡})$ / $𝜋_{𝜃_{old}} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡})$ is the probability of generating token $𝑜_{𝑖, 𝑡}$ under current/old policy ( $𝜋_{𝜃}$ / $𝜋_{𝜃_{old}}$ )
${\hat{𝐴}}_{𝑖, 𝑡}$ is the advantage of the $𝑡$ -th token in the $𝑖$ -th response

GRPO maximizes the expected total reward of the group. It:

increases probability of high-advantage responses
decreases probability of low-advantage responses
keeps the policy update within a trust region

Probability Ratio $𝑟_{𝑖, 𝑡} (𝜃)$

The probability ratio measures how much the current policy differs from the old policy for each token:

𝑟_{𝑖, 𝑡} (𝜃) = \frac{𝜋_{𝜃} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡})}{𝜋_{𝜃_{old}} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡})}

where:

$𝜋_{𝜃} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡})$ : probability of generating token $𝑜_{𝑖, 𝑡}$ under current policy ( $\in [0, 1]$ )
$𝜋_{𝜃_{old}} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡})$ : probability under old policy ( $\in [0, 1]$ )
$𝑜_{𝑖, < 𝑡}$ : sequence of tokens before position $𝑡$ in response $𝑖$ (token sequence of length $𝑡 - 1$ )

Advantage ${\hat{𝐴}}_{𝑖, 𝑡}$

The advantage of the $𝑖$ -th response is calculated by normalizing the group-level rewards ${𝑅_{𝑖}}_{𝑖 = 1}^{𝐺}$ :

{\hat{𝐴}}_{𝑖, 𝑡} = \frac{𝑅_{𝑖} - mean ({𝑅_{𝑖}}_{𝑖 = 1}^{𝐺})}{std ({𝑅_{𝑖}}_{𝑖 = 1}^{𝐺})}

where:

$𝑅_{𝑖}$ : reward for the $𝑖$ -th response ( $\in {0, 1}$ )
$mean ({𝑅_{𝑖}}_{𝑖 = 1}^{𝐺}) = \frac{1}{𝐺} \sum_{𝑗 = 1}^{𝐺} 𝑅_{𝑗}$ : average reward across group
$std ({𝑅_{𝑖}}_{𝑖 = 1}^{𝐺})$ : standard deviation of group rewards
${\hat{𝐴}}_{𝑖, 𝑡}$ : normalized advantage for token $𝑡$ in response $𝑖$

Note that ${\hat{𝐴}}_{𝑖, 𝑡}$ is computed at the response level (using response reward $𝑅_{𝑖}$ ), so for all tokens in the same response, ${\hat{𝐴}}_{𝑖, 𝑡}$ is the same.

The Clipping Mechanism

The core of GRPO (inherited from PPO [2]) is the clipped objective:

min (𝑟_{𝑖, 𝑡} (𝜃) {\hat{𝐴}}_{𝑖, 𝑡}, clip (𝑟_{𝑖, 𝑡} (𝜃), 1 - 𝜀, 1 + 𝜀) {\hat{𝐴}}_{𝑖, 𝑡})

Recall from the probability ratio and advantage definitions that:

\begin{aligned} 𝑟_{𝑖, 𝑡} (𝜃) & \in [0, + \infty) \\ {\hat{𝐴}}_{𝑖, 𝑡} & \in {-, +} \end{aligned}

The behavior depends on the sign of the advantage. The following table shows the raw analysis:

${\hat{𝐴}}_{𝑖, 𝑡}$	$𝑟_{𝑖, 𝑡} (𝜃)$	Objective	Clip?	Prevent what?
$> 0$	$(0, 1 - 𝜀)$	$(1 - 𝜀) {\hat{𝐴}}_{𝑖, 𝑡}$	Yes	no/less advantage on positive
$> 0$	$[1 - 𝜀, 1 + 𝜀]$	$𝑟_{𝑖, 𝑡} (𝜃) {\hat{𝐴}}_{𝑖, 𝑡}$	No	normal
$> 0$	$(1 + 𝜀, + \infty)$	$(1 + 𝜀) {\hat{𝐴}}_{𝑖, 𝑡}$	Yes	over advantage on positive
$< 0$	$(0, 1 - 𝜀)$	$(1 - 𝜀) {\hat{𝐴}}_{𝑖, 𝑡}$	Yes	no/less penalty on negative
$< 0$	$[1 - 𝜀, 1 + 𝜀]$	$𝑟_{𝑖, 𝑡} (𝜃) {\hat{𝐴}}_{𝑖, 𝑡}$	No	normal
$< 0$	$(1 + 𝜀, + \infty)$	$(1 + 𝜀) {\hat{𝐴}}_{𝑖, 𝑡}$	Yes	over penalty on negative

Taking the $min$ in the clipped objective simplifies the behavior. When ${\hat{𝐴}}_{𝑖, 𝑡} > 0$ , the $min$ effectively computes $min (𝑟_{𝑖, 𝑡} (𝜃), clip (𝑟_{𝑖, 𝑡} (𝜃), 1 - 𝜀, 1 + 𝜀))$ . When ${\hat{𝐴}}_{𝑖, 𝑡} < 0$ , it becomes $max (𝑟_{𝑖, 𝑡} (𝜃), clip (𝑟_{𝑖, 𝑡} (𝜃), 1 - 𝜀, 1 + 𝜀))$ :

${\hat{𝐴}}_{𝑖, 𝑡}$	Effective	$𝑟_{𝑖, 𝑡} (𝜃)$	Result	Prevent what?
$> 0$	$min (𝑟, clip (𝑟, 1 - 𝜀, 1 + 𝜀))$	$(0, 1 - 𝜀)$	$𝑟_{𝑖, 𝑡} (𝜃)$	already safe
$> 0$	$min (𝑟, clip (𝑟, 1 - 𝜀, 1 + 𝜀))$	$(1 - 𝜀, 1 + 𝜀)$	$𝑟_{𝑖, 𝑡} (𝜃)$	trust region
$> 0$	$min (𝑟, clip (𝑟, 1 - 𝜀, 1 + 𝜀))$	$(1 + 𝜀, + \infty)$	$1 + 𝜀$	over-increase
$< 0$	$max (𝑟, clip (𝑟, 1 - 𝜀, 1 + 𝜀))$	$(0, 1 - 𝜀)$	$1 - 𝜀$	over-decrease
$< 0$	$max (𝑟, clip (𝑟, 1 - 𝜀, 1 + 𝜀))$	$(1 - 𝜀, 1 + 𝜀)$	$𝑟_{𝑖, 𝑡} (𝜃)$	trust region
$< 0$	$max (𝑟, clip (𝑟, 1 - 𝜀, 1 + 𝜀))$	$(1 + 𝜀, + \infty)$	$𝑟_{𝑖, 𝑡} (𝜃)$	self-penalizing

Intuition:

If ${\hat{𝐴}}_{𝑖, 𝑡} > 0$ and $𝑟_{𝑖, 𝑡} (𝜃) > 1 + 𝜀$ : the model already has higher advantage on this case. To prevent over-increase, clip $𝑟_{𝑖, 𝑡} (𝜃)$ to $1 + 𝜀$ .
If ${\hat{𝐴}}_{𝑖, 𝑡} < 0$ and $𝑟_{𝑖, 𝑡} (𝜃) < 1 - 𝜀$ : the model already has lower advantage on this case. To prevent over-decrease, clip $𝑟_{𝑖, 𝑡} (𝜃)$ to $1 - 𝜀$ .

As stated in the original PPO paper: “We only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse.”

Example

Let’s walk through a concrete example to illustrate how GRPO works in practice.

Setup

Question: $𝑞 = "What is the capital of France?"$
Group size: $𝐺 = 4$
Clipping parameter: $𝜀 = 0.2$

Step 1: Group Sampling

The old policy $𝜋_{𝜃_{old}}$ generates $𝐺 = 4$ responses for the question with varying lengths:

Response	Generated Text	Length
$𝑜_{1}$	“The capital of France is Paris.”	7 tokens
$𝑜_{2}$	“Well, the capital city of France, which is Paris.”	12 tokens
$𝑜_{3}$	“Paris, France.”	4 tokens
$𝑜_{4}$	“France’s capital city is the beautiful Paris, located in Europe.”	12 tokens

Step 2: Reward and Advantage

Each response receives a binary reward based on correctness, then we compute group statistics and advantages:

Response	Reward	Explanation
$𝑜_{1}$	$𝑅_{1} = 1$	Correct
$𝑜_{2}$	$𝑅_{2} = 1$	Correct
$𝑜_{3}$	$𝑅_{3} = 1$	Correct
$𝑜_{4}$	$𝑅_{4} = 0$	Incorrect

Compute the group-level statistics for advantage estimation:

$mean ({𝑅_{𝑖}}_{𝑖 = 1}^{4}) = \frac{1 + 1 + 1 + 0}{4} = 0.75$

$std ({𝑅_{𝑖}}_{𝑖 = 1}^{4}) = \sqrt{\frac{\sum_{𝑖 = 1}^{4} {(𝑅_{𝑖} - 0.75)}^{2}}{4}} = 0.433$

Using the advantage formula, compute the advantage for each response:

Response	Advantage	Result
$𝑜_{1}$	${\hat{𝐴}}_{1, 𝑡} = \frac{1 - 0.75}{0.433}$	$\approx 0.577$
$𝑜_{2}$	${\hat{𝐴}}_{2, 𝑡} = \frac{1 - 0.75}{0.433}$	$\approx 0.577$
$𝑜_{3}$	${\hat{𝐴}}_{3, 𝑡} = \frac{1 - 0.75}{0.433}$	$\approx 0.577$
$𝑜_{4}$	${\hat{𝐴}}_{4, 𝑡} = \frac{0 - 0.75}{0.433}$	$\approx - 1.732$

Note that ${\hat{𝐴}}_{𝑖, 𝑡}$ is the same for all tokens $𝑡$ within response $𝑖$ .

Step 3: Per-Token Probability Ratios

For each token $(𝑖, 𝑡)$ , we need both $𝜋_{𝜃_{old}} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡})$ and $𝜋_{𝜃} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡})$ to compute the probability ratio.

Response 1: $𝑜_{1} = [The, capital, of, France, is, Paris, .]$

Pos	Token	Context	Old Policy	New Policy	Ratio	Clipped
1	“The”	$𝑞$	$0.30$	$0.35$	$1.167$	$1.167$
2	“capital”	$𝑞$ , “The”	$0.25$	$0.28$	$1.120$	$1.120$
3	“of”	$𝑞$ , “The”, “capital”	$0.40$	$0.45$	$1.125$	$1.125$
4	“France”	$𝑞$ , …, “of”	$0.35$	$0.32$	$0.914$	$0.914$
5	“is”	$𝑞$ , …, “France”	$0.45$	$0.50$	$1.111$	$1.111$
6	“Paris”	$𝑞$ , …, “is”	$0.60$	$0.65$	$1.083$	$1.083$
7	“.”	$𝑞$ , …, “Paris”	$0.80$	$0.85$	$1.063$	$1.063$

Response 2: $𝑜_{2} = [Well,,, the, capital, city, of, France,,, which, is, Paris, .]$

Pos	Token	Context	Old Policy	New Policy	Ratio	Clipped
1	“Well”	$𝑞$	$0.15$	$0.12$	$0.800$	$0.800$
2	“,”	$𝑞$ , “Well”	$0.40$	$0.35$	$0.875$	$0.875$
3	“the”	$𝑞$ , “Well”, “,”	$0.50$	$0.55$	$1.100$	$1.100$
4	“capital”	$𝑞$ , …, “the”	$0.45$	$0.48$	$1.067$	$1.067$
5	“city”	$𝑞$ , …, “capital”	$0.30$	$0.25$	$0.833$	$0.833$
6	“of”	$𝑞$ , …, “city”	$0.40$	$0.45$	$1.125$	$1.125$
7	“France”	$𝑞$ , …, “of”	$0.35$	$0.40$	$1.143$	$1.143$
8	“,”	$𝑞$ , …, “France”	$0.25$	$0.20$	$0.800$	$0.800$
9	“which”	$𝑞$ , …, “,”	$0.20$	$0.15$	$0.750$	$0.800$
10	“is”	$𝑞$ , …, “which”	$0.60$	$0.65$	$1.083$	$1.083$
11	“Paris”	$𝑞$ , …, “is”	$0.70$	$0.75$	$1.071$	$1.071$
12	“.”	$𝑞$ , …, “Paris”	$0.80$	$0.85$	$1.063$	$1.063$

Response 3: $𝑜_{3} = [Paris,,, France, .]$

Pos	Token	Context	Old Policy	New Policy	Ratio	Clipped
1	“Paris”	$𝑞$	$0.20$	$0.25$	$1.250$	$1.200$
2	“,”	$𝑞$ , “Paris”	$0.30$	$0.35$	$1.167$	$1.167$
3	“France”	$𝑞$ , “Paris”, “,”	$0.25$	$0.30$	$1.200$	$1.200$
4	“.”	$𝑞$ , …, “France”	$0.70$	$0.75$	$1.071$	$1.071$

Note how “Paris” in Response 3 has $𝑟_{3, 1} = 1.250 > 1 + 𝜀 = 1.2$ , so it gets clipped to $1.200$ .

Response 4: $𝑜_{4} = [France's, capital, city, is, the, beautiful, Paris,,, located, in, Europe, .]$

Pos	Token	Context	Old Policy	New Policy	Ratio	Clipped
1	“France’s”	$𝑞$	$0.10$	$0.08$	$0.800$	$0.800$
2	“capital”	$𝑞$ , “France’s”	$0.25$	$0.30$	$1.200$	$1.200$
3	“city”	$𝑞$ , …, “capital”	$0.15$	$0.18$	$1.200$	$1.200$
4	“is”	$𝑞$ , …, “city”	$0.40$	$0.45$	$1.125$	$1.125$
5	“the”	$𝑞$ , …, “is”	$0.35$	$0.40$	$1.143$	$1.143$
6	“beautiful”	$𝑞$ , …, “the”	$0.10$	$0.08$	$0.800$	$0.800$
7	“Paris”	$𝑞$ , …, “beautiful”	$0.60$	$0.65$	$1.083$	$1.083$
8	“,”	$𝑞$ , …, “Paris”	$0.20$	$0.18$	$0.900$	$0.900$
9	“located”	$𝑞$ , …, “,”	$0.15$	$0.12$	$0.800$	$0.800$
10	“in”	$𝑞$ , …, “located”	$0.45$	$0.50$	$1.111$	$1.111$
11	“Europe”	$𝑞$ , …, “in”	$0.25$	$0.20$	$0.800$	$0.800$
12	“.”	$𝑞$ , …, “Europe”	$0.75$	$0.80$	$1.067$	$1.067$

Step 4: Objective Terms

For each token $(𝑖, 𝑡)$ , compute the clipped objective term:

${obj}_{𝑖, 𝑡} = min (𝑟_{𝑖, 𝑡} (𝜃) {\hat{𝐴}}_{𝑖, 𝑡}, 𝑟_{𝑖, 𝑡}^{clipped} {\hat{𝐴}}_{𝑖, 𝑡})$

Recall our advantages from Step 2:

${\hat{𝐴}}_{1, 𝑡} = 0.577$ for all $𝑡$ in response 1
${\hat{𝐴}}_{2, 𝑡} = 0.577$ for all $𝑡$ in response 2
${\hat{𝐴}}_{3, 𝑡} = 0.577$ for all $𝑡$ in response 3
${\hat{𝐴}}_{4, 𝑡} = - 1.732$ for all $𝑡$ in response 4

Response 1 objective terms ( ${\hat{𝐴}}_{1, 𝑡} = 0.577 > 0$ ):

${obj}_{1, 1} = min (1.167 \times 0.577, 1.167 \times 0.577) = 0.673$
${obj}_{1, 2} = min (1.120 \times 0.577, 1.120 \times 0.577) = 0.646$
${obj}_{1, 3} = min (1.125 \times 0.577, 1.125 \times 0.577) = 0.649$
${obj}_{1, 4} = min (0.914 \times 0.577, 0.914 \times 0.577) = 0.527$
${obj}_{1, 5} = min (1.111 \times 0.577, 1.111 \times 0.577) = 0.641$
${obj}_{1, 6} = min (1.083 \times 0.577, 1.083 \times 0.577) = 0.625$
${obj}_{1, 7} = min (1.063 \times 0.577, 1.063 \times 0.577) = 0.613$

Response 2 objective terms ( ${\hat{𝐴}}_{2, 𝑡} = 0.577 > 0$ ):

${obj}_{2, 1} = min (0.800 \times 0.577, 0.800 \times 0.577) = 0.462$
${obj}_{2, 2} = min (0.875 \times 0.577, 0.875 \times 0.577) = 0.505$
${obj}_{2, 3} = min (1.100 \times 0.577, 1.100 \times 0.577) = 0.635$
${obj}_{2, 4} = min (1.067 \times 0.577, 1.067 \times 0.577) = 0.616$
${obj}_{2, 5} = min (0.833 \times 0.577, 0.833 \times 0.577) = 0.481$
${obj}_{2, 6} = min (1.125 \times 0.577, 1.125 \times 0.577) = 0.649$
${obj}_{2, 7} = min (1.143 \times 0.577, 1.143 \times 0.577) = 0.660$
${obj}_{2, 8} = min (0.800 \times 0.577, 0.800 \times 0.577) = 0.462$
${obj}_{2, 9} = min (0.750 \times 0.577, 0.800 \times 0.577) = 0.433$
${obj}_{2, 10} = min (1.083 \times 0.577, 1.083 \times 0.577) = 0.625$
${obj}_{2, 11} = min (1.071 \times 0.577, 1.071 \times 0.577) = 0.618$
${obj}_{2, 12} = min (1.063 \times 0.577, 1.063 \times 0.577) = 0.613$

Response 3 objective terms ( ${\hat{𝐴}}_{3, 𝑡} = 0.577 > 0$ ):

${obj}_{3, 1} = min (1.250 \times 0.577, 1.200 \times 0.577) = min (0.721, 0.692) = 0.692$
${obj}_{3, 2} = min (1.167 \times 0.577, 1.167 \times 0.577) = 0.673$
${obj}_{3, 3} = min (1.200 \times 0.577, 1.200 \times 0.577) = 0.692$
${obj}_{3, 4} = min (1.071 \times 0.577, 1.071 \times 0.577) = 0.618$

Response 4 objective terms ( ${\hat{𝐴}}_{4, 𝑡} = - 1.732 < 0$ ):

${obj}_{4, 1} = min (0.800 \times (- 1.732), 0.800 \times (- 1.732)) = - 1.386$
${obj}_{4, 2} = min (1.200 \times (- 1.732), 1.200 \times (- 1.732)) = - 2.078$
${obj}_{4, 3} = min (1.200 \times (- 1.732), 1.200 \times (- 1.732)) = - 2.078$
${obj}_{4, 4} = min (1.125 \times (- 1.732), 1.125 \times (- 1.732)) = - 1.949$
${obj}_{4, 5} = min (1.143 \times (- 1.732), 1.143 \times (- 1.732)) = - 1.980$
${obj}_{4, 6} = min (0.800 \times (- 1.732), 0.800 \times (- 1.732)) = - 1.386$
${obj}_{4, 7} = min (1.083 \times (- 1.732), 1.083 \times (- 1.732)) = - 1.876$
${obj}_{4, 8} = min (0.900 \times (- 1.732), 0.900 \times (- 1.732)) = - 1.559$
${obj}_{4, 9} = min (0.800 \times (- 1.732), 0.800 \times (- 1.732)) = - 1.386$
${obj}_{4, 10} = min (1.111 \times (- 1.732), 1.111 \times (- 1.732)) = - 1.924$
${obj}_{4, 11} = min (0.800 \times (- 1.732), 0.800 \times (- 1.732)) = - 1.386$
${obj}_{4, 12} = min (1.067 \times (- 1.732), 1.067 \times (- 1.732)) = - 1.848$

Step 5: Final Objective

Per-response averages:

Response 1: $\frac{1}{7} \sum_{𝑡 = 1}^{7} {obj}_{1, 𝑡} = \frac{1}{7} (0.673 + 0.646 + \dots) = 0.625$
Response 2: $\frac{1}{12} \sum_{𝑡 = 1}^{12} {obj}_{2, 𝑡} = \frac{1}{12} (0.462 + 0.505 + \dots) = 0.563$
Response 3: $\frac{1}{4} \sum_{𝑡 = 1}^{4} {obj}_{3, 𝑡} = \frac{1}{4} (0.692 + 0.673 + \dots) = 0.669$
Response 4: $\frac{1}{12} \sum_{𝑡 = 1}^{12} {obj}_{4, 𝑡} = \frac{1}{12} (- 1.386 - 2.078 - \dots) = - 1.736$

Final GRPO objective (group average):

$\begin{aligned} {𝒥︀}_{GRPO (𝜃)} & = \frac{1}{𝐺} \sum_{𝑖 = 1}^{𝐺} \frac{1}{| 𝑜_{𝑖} |} \sum_{𝑡 = 1}^{| 𝑜_{𝑖} |} {obj}_{𝑖, 𝑡} \\ = \frac{1}{4} (0.625 + 0.563 + 0.669 + (- 1.736)) = 0.030 \end{aligned}$

Follow-ups and Variants

GRPO in DeepSeek R1

DeepSeek R1 [3] modifies the original GRPO objective which includes a length normalization term $\frac{1}{| 𝑜_{𝑖} |}$ and a KL divergence penalty:

\begin{aligned} 𝒥︀ (𝜃) & = 𝔼_{{(𝑞, 𝑎) \sim 𝒟︀, {𝑜_{𝑖}}_{𝑖 = 1}^{𝐺} \sim 𝜋_{𝜃_{old}} (\cdot | 𝑞)}} \\ [\frac{1}{𝐺} \sum_{𝑖 = 1}^{𝐺} \frac{1}{| 𝑜_{𝑖} |} \sum_{𝑡 = 1}^{| 𝑜_{𝑖} |} [min (𝑟_{𝑖, 𝑡} (𝜃) {\hat{𝐴}}_{𝑖, 𝑡}, clip (𝑟_{𝑖, 𝑡} (𝜃), 1 - 𝜀, 1 + 𝜀) {\hat{𝐴}}_{𝑖, 𝑡}) - 𝛽 𝐷_{KL}]] \end{aligned}

DeepSeek R1 removes the length normalization term $\frac{1}{| 𝑜_{𝑖} |}$ :

\begin{aligned} 𝒥︀ (𝜃) & = 𝔼_{{(𝑞, 𝑎) \sim 𝒟︀, {𝑜_{𝑖}}_{𝑖 = 1}^{𝐺} \sim 𝜋_{𝜃_{old}} (\cdot | 𝑞)}} \\ [\frac{1}{𝐺} \sum_{𝑖 = 1}^{𝐺} \sum_{𝑡 = 1}^{| 𝑜_{𝑖} |} [min (𝑟_{𝑖, 𝑡} (𝜃) {\hat{𝐴}}_{𝑖, 𝑡}, clip (𝑟_{𝑖, 𝑡} (𝜃), 1 - 𝜀, 1 + 𝜀) {\hat{𝐴}}_{𝑖, 𝑡}) - 𝛽 𝐷_{KL}]] \end{aligned}

Dr. GRPO

Dr. GRPO [4] (GRPO Done Right, without bias) removes both the length normalization and the standard deviation normalization:

\begin{aligned} {𝒥︀}_{Dr. GRPO (𝜃)} & = 𝔼_{{(𝑞, 𝑎) \sim 𝒟︀, {𝑜_{𝑖}}_{𝑖 = 1}^{𝐺} \sim 𝜋_{𝜃_{old}} (\cdot | 𝑞)}} \\ [\frac{1}{𝐺} \sum_{𝑖 = 1}^{𝐺} \sum_{𝑡 = 1}^{| 𝑜_{𝑖} |} min (𝑟_{𝑖, 𝑡} (𝜃) {\hat{𝐴}}_{𝑖, 𝑡}, clip (𝑟_{𝑖, 𝑡} (𝜃), 1 - 𝜀, 1 + 𝜀) {\hat{𝐴}}_{𝑖, 𝑡})], \end{aligned}

where ${\hat{𝐴}}_{𝑖, 𝑡} = 𝑅_{𝑖} - mean ({𝑅_{𝑖}}_{𝑖 = 1}^{𝐺})$ (no std normalization compared to the original GRPO advantage).

DAPO

Decouple Clip and Dynamic sAmpling Policy Optimization (DAPO) [5] introduces several modifications:

\begin{aligned} {𝒥︀}_{DAPO (𝜃)} & = 𝔼_{{(𝑞, 𝑎) \sim 𝒟︀, {𝑜_{𝑖}}_{𝑖 = 1}^{𝐺} \sim 𝜋_{𝜃_{old}} (\cdot | 𝑞)}} \\ [\frac{1}{\sum_{𝑖 = 1}^{𝐺} | 𝑜_{𝑖} |} \sum_{𝑖 = 1}^{𝐺} \sum_{𝑡 = 1}^{| 𝑜_{𝑖} |} min (𝑟_{𝑖, 𝑡} (𝜃) {\hat{𝐴}}_{𝑖, 𝑡}, clip (𝑟_{𝑖, 𝑡} (𝜃), 1 - 𝜀_{low}, 1 + 𝜀_{high}) {\hat{𝐴}}_{𝑖, 𝑡})] \\ s.t. 0 < | {𝑜_{𝑖} | is_equivalent (𝑎, 𝑜_{𝑖})} | < 𝐺, \end{aligned}

Key innovations:

Clip-Higher: asymmetric clipping with separate $𝜀_{low}$ and $𝜀_{high}$ , raising the ceiling
Dynamic sampling: over-sample and filter out prompts with the accuracy equal to 1 and 0
Token-level loss: rebalancing with token-level policy gradient loss

GPG

Group Policy Gradient (GPG) [6] uses log-probabilities directly instead of importance sampling ratios:

\begin{aligned} {𝒥︀}_{GPG (𝜃)} = & 𝔼_{{(𝑞, 𝑎) \sim 𝒟︀, {𝑜_{𝑖}}_{𝑖 = 1}^{𝐺}}} \\ [\frac{1}{\sum_{𝑖 = 1}^{𝐺} | 𝑜_{𝑖} |} \sum_{𝑖 = 1}^{𝐺} \sum_{𝑡 = 1}^{| 𝑜_{𝑖} |} (log 𝜋_{𝜃} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡}) {\hat{𝐴}}_{𝑖, 𝑡})], \end{aligned}

where ${\hat{𝐴}}_{𝑖, 𝑡} = \frac{𝑅_{𝑖} - mean ({𝑅_{𝑖}}_{𝑖 = 1}^{𝐺})}{𝐹_{norm}}$ with $𝐹_{norm}$ being an optional normalization technique.

CISPO

Clipped IS-weight Policy Optimization (CISPO) [7] combines importance sampling weights with log-probability gradients:

\begin{aligned} {𝒥︀}_{CISPO (𝜃)} = & 𝔼_{{(𝑞, 𝑎) \sim 𝒟︀, {𝑜_{𝑖}}_{𝑖 = 1}^{𝐺} \sim 𝜋_{𝜃_{old}} (\cdot | 𝑞)}} \\ [\frac{1}{\sum_{𝑖 = 1}^{𝐺} | 𝑜_{𝑖} |} \sum_{𝑖 = 1}^{𝐺} \sum_{𝑡 = 1}^{| 𝑜_{𝑖} |} sg ({\hat{𝑟}}_{𝑖, 𝑡} (𝜃)) {\hat{𝐴}}_{𝑖, 𝑡} log 𝜋_{𝜃} (𝑜_{𝑖, 𝑡} | 𝑞, 𝑜_{𝑖, < 𝑡})], \end{aligned}

where ${\hat{𝑟}}_{𝑖, 𝑡} (𝜃) = clip (𝑟_{𝑖, 𝑡} (𝜃), 1 - 𝜀_{low}^{IS}, 1 + 𝜀_{high}^{IS})$ is the clipped IS weight and $sg (\cdot)$ denotes stop-gradient.

C3PO

Constrained Contextual Computation Policy Optimization (C3PO) [8] introduces a training token budget constraint:

\begin{aligned} {ℒ︀}_{C3PO (𝜃)} & = 𝔼_{{{𝑞, 𝑎}_{𝑙 = 1}^{𝐿} \sim 𝒟︀, {𝑦_{𝑖}}_{𝑖 = 1}^{𝐾} \sim 𝜋_{old (\cdot | 𝑞)}}} \\ [\frac{1}{Φ} \sum_{𝑖 = 1}^{𝑆} \sum_{𝑡 = 1}^{| 𝑦_{𝑖} |} 𝕀 [𝑦_{𝑖, 𝑡} \in Ψ] (min (𝑟_{𝑖, 𝑡} (𝜃) 𝐴_{𝑖, 𝑡}, clip (𝑟_{𝑖, 𝑡} (𝜃), 1 - 𝜀, 1 + 𝜀) 𝐴_{𝑖, 𝑡}) - 𝛽 𝐷_{KL}))], \end{aligned}

subject to $| Ψ | = Φ$ , where:

$Ψ = {(𝑦_{1}, 𝑦_{2}, \dots, 𝑦_{𝑁}) | 𝑦_{𝑖} \in ℬ︀} s.t. 𝑁 \leq | ℬ︀ |, \sum_{𝑗 = 1}^{𝑁 - 1} | 𝑦_{𝑗} | < Φ, \sum_{𝑗 = 1}^{𝑁} | 𝑦_{𝑗} | \geq Φ$

and:

$Φ$ is the training token budget
$Ψ$ is the selected tokens by custom sampling strategy
$𝑆$ is the selected responses for training
$𝑦_{𝑖}$ is the $𝑖$ -th response

Value-Based Variants

PPO

A response trajectory $𝜏 = (𝑠_{0}, 𝑎_{0}, 𝑟_{1}, \dots, 𝑠_{𝑡}, 𝑎_{𝑡}, 𝑟_{𝑡 + 1}, \dots, 𝑠_{𝑇 - 1}, 𝑎_{𝑇 - 1}, 𝑟_{𝑇}, 𝑠_{𝑇})$ . PPO [2] optimizes the total rewards:

{𝒥︀}_{PPO (𝜃)} = 𝔼_{𝑡, 𝑠_{𝑡}, 𝑎_{𝑡} \sim 𝜋_{𝜃_{old}}} [min (𝑟_{𝑡} {\hat{𝐴}}_{𝑡}, clip (𝑟_{𝑡}, 1 - 𝜀, 1 + 𝜀) {\hat{𝐴}}_{𝑡})]

where the advantage is estimated via Generalized Advantage Estimation (GAE):

\begin{aligned} {\hat{𝐴}}_{𝑡} & = 𝛿_{𝑡} + (𝛾 𝜆) 𝛿_{𝑡 + 1} + {(𝛾 𝜆)}^{2} 𝛿_{𝑡 + 2} + \dots + {(𝛾 𝜆)}^{𝑇 - 𝑡 - 1} 𝛿_{𝑇 - 1} \\ 𝛿_{𝑡} & = 𝑟_{𝑡 + 1} + 𝛾 𝑉 (𝑠_{𝑡 + 1}) - 𝑉 (𝑠_{𝑡}) \end{aligned}

Here $𝛿_{𝑡}$ is the Temporal Difference (TD) error between the estimated value of the next state $𝑟_{𝑡 + 1} + 𝛾 𝑉 (𝑠_{𝑡 + 1})$ and the estimated value of the current state $𝑉 (𝑠_{𝑡})$ .

VC-PPO

Value-Calibrated PPO (VC-PPO) [9] uses the same PPO objective, and addresses the value initialization bias through:

Value-Pretraining: addressing the value initialization bias by value pretraining
Decoupled-GAE: improving in-training value estimate with decoupled-GAE

VAPO

Value-model-based Augmented PPO (VAPO) [10] optimizes:

{ℒ︀}_{VAPO (𝜃)} = [\frac{1}{\sum_{𝑖 = 1}^{𝐺} | 𝑜_{𝑖} |} \sum_{𝑖 = 1}^{𝐺} \sum_{𝑡 = 1}^{| 𝑜_{𝑖} |} min (𝑟_{𝑖, 𝑡} (𝜃) {\hat{𝐴}}_{𝑖, 𝑡}, clip (𝑟_{𝑖, 𝑡} (𝜃), 1 - 𝜀_{low}, 1 + 𝜀_{high}) {\hat{𝐴}}_{𝑖, 𝑡})]

Key innovations:

Length-Adaptive GAE: address the inconsistency across sequences of varying lengths
Token-level policy gradient loss
Dealing with sparsity of reward signal in verifier-based tasks:
- Clip-higher
- Positive example LM loss: additional negative log-likelihood (NLL) loss for the correct outcomes
- Group-sampling: sample discriminative positive and negative samples within the same prompt

T-PPO

Truncated Proximal Policy Optimization (T-PPO) [11] optimizes:

\begin{aligned} {𝒥︀}_{T-PPO (𝜃)} & = 𝔼_{𝑡, 𝑠_{𝑡}, 𝑎_{𝑡} \sim 𝜋_{𝜃_{old}}} \\ [min (\frac{𝜋_{𝜃} (𝑎_{𝑡} | 𝑠_{𝑡})}{𝜋_{𝜃_{old}} (𝑎_{𝑡} | 𝑠_{𝑡})} {\hat{𝐴}}_{𝑡}, clip (\frac{𝜋_{𝜃} (𝑎_{𝑡} | 𝑠_{𝑡})}{𝜋_{𝜃_{old}} (𝑎_{𝑡} | 𝑠_{𝑡})}, 1 - 𝜀_{low}, 1 + 𝜀_{high}) {\hat{𝐴}}_{𝑡})] \end{aligned}

Key innovations:

Extended GAE: improved advantage estimation
Token filtering: if some sequences reach an ending condition, these sequences are removed in the next training step

References

[1] Z. Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” arXiv preprint arXiv:2402.03300, 2024.
[2] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” arXiv preprint arXiv:1707.06347, 2017.
[3] DeepSeek-AI et al., “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” Nature, vol. 645, pp. 633–638, 2025.
[4] Z. Liu et al., “Understanding R1-Zero-Like Training: A Critical Perspective,” arXiv preprint arXiv:2503.20783, 2025.
[5] Q. Yu et al., “DAPO: An Open-Source LLM Reinforcement Learning System at Scale,” arXiv preprint arXiv:2503.14476, 2025.
[6] X. Chu, H. Huang, X. Zhang, F. Wei, and Y. Wang, “GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning,” arXiv preprint arXiv:2504.02546, 2025.
[7] MiniMax et al., “MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention,” arXiv preprint arXiv:2506.13585, 2025.
[8] Ling Team et al., “Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs,” arXiv preprint arXiv:2506.14731, 2025.
[9] Y. Yuan, Y. Yue, R. Zhu, T. Fan, and L. Yan, “What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret,” arXiv preprint arXiv:2503.01491, 2025.
[10] Y. Yue et al., “VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks,” arXiv preprint arXiv:2504.05118, 2025.
[11] T. Fan et al., “Truncated Proximal Policy Optimization,” arXiv preprint arXiv:2506.15050, 2025.

Objective Function

Probability Ratio 𝑟𝑖,𝑡(𝜃)

Advantage 𝐴̂𝑖,𝑡

The Clipping Mechanism

Example

Setup

Step 1: Group Sampling

Step 2: Reward and Advantage

Step 3: Per-Token Probability Ratios

Step 4: Objective Terms

Step 5: Final Objective

Follow-ups and Variants

GRPO in DeepSeek R1

Dr. GRPO

DAPO

GPG

CISPO

C3PO

Value-Based Variants

PPO

VC-PPO

VAPO

T-PPO

References

Probability Ratio $𝑟_{𝑖, 𝑡} (𝜃)$

Advantage ${\hat{𝐴}}_{𝑖, 𝑡}$