Background: Proximal Policy Optimization (PPO)

PPO introduces a clipped surrogate objective for policy optimization. By constraining the policy updates within a proximal region of the previous policy using clip, PPO stabilizes training and improves sample efficiency. Specifically, PPO updates the policy by maximizing the following objective:

$${𝒥}_{\text{PPO}}\left(𝜃\right)={𝔼}_{\left(𝑞,𝑎\right)\sim 𝒟}\left[\min \left(\frac{{𝜋}_{𝜃\left({𝑜}_{𝑡}|𝑞,{𝑜}_{<𝑡}\right)}}{{𝜋}_{{𝜃}_{\text{old}}}\left({𝑜}_{𝑡}|𝑞,{𝑜}_{<𝑡}\right)}{\widehat{𝐴}}_{𝑡},\text{ clip }\left(\frac{{𝜋}_{𝜃\left({𝑜}_{𝑡}|𝑞,{𝑜}_{<𝑡}\right)}}{{𝜋}_{{𝜃}_{\text{old}}}\left({𝑜}_{𝑡}|𝑞,{𝑜}_{<𝑡}\right)},1-𝜀,1+𝜀\right){\widehat{𝐴}}_{𝑡}\right)\right]$$

where $\left(𝑞,𝑎\right)$ is a question-answer pair from the data distribution $𝒟$, $𝜀$ is the clipping range of importance sampling ratio, and ${\widehat{𝐴}}_{𝑡}$ is an estimator of the advantage at time step $𝑡$. Given the value function $𝑉$ and the reward function $𝑅$, ${\widehat{𝐴}}_{𝑡}$ is computed using the Generalized Advantage Estimation (GAE):

$${\widehat{𝐴}}_{𝑡}^{\text{GAE}}\left(𝛾,𝜆\right)=\sum _{𝑙=0}^{\infty }{\left(𝛾𝜆\right)}^{𝑙}{𝛿}_{𝑡+𝑙},$$

where

$${𝛿}_{𝑙}={𝑅}_{𝑙}+𝛾𝑉\left({𝑠}_{𝑙+1}\right)-𝑉\left({𝑠}_{𝑙}\right),0\le 𝛾,𝜆\le 1.$$

Group Relative Policy Optimization (GRPO)

Compared to PPO, GRPO eliminates the value function and estimates the advantage in a group-relative manner.

Objective Function

Similar to PPO, GRPO maximizes a clipped objective, together with a directly imposed KL penalty term:

$${𝒥}_{\text{GRPO}}\left(𝜃\right)={𝔼}_{\left(𝑞,𝑎\right)\sim 𝒟}\frac{1}{𝐺}\sum _{𝑖=1}^{𝐺}\frac{1}{\left|{𝑜}_{𝑖}\right|}\sum _{𝑡=1}^{\left|{𝑜}_{𝑖}\right|}\left[\min \left({𝑟}_{𝑖,𝑡}\left(𝜃\right){\widehat{𝐴}}_{𝑖,𝑡},\text{ clip }\left({𝑟}_{𝑖,𝑡}\left(𝜃\right),1-𝜀,1+𝜀\right){\widehat{𝐴}}_{𝑖,𝑡}\right)-𝛽{𝐷}_{\text{ KL}}\left({𝜋}_{𝜃}\Vert {𝜋}_{\text{ ref}}\right)\right]$$

where:

• ${𝑟}_{𝑖,𝑡}\left(𝜃\right)=\frac{{𝜋}_{𝜃}\left({𝑜}_{𝑖,𝑡}|𝑞,{𝑜}_{𝑖,<𝑡}\right)}{{𝜋}_{{𝜃}_{\text{old}}}\left({𝑜}_{𝑖,𝑡}|𝑞,{𝑜}_{𝑖,<𝑡}\right)}$ is the importance sampling ratio for the $𝑖$-th response at time step $𝑡$
• ${\widehat{𝐴}}_{𝑖,𝑡}$ is the advantage for the $𝑖$-th response at time step $𝑡$
• $𝛽$ is the KL penalty coefficient
• ${𝐷}_{\text{KL}}\left({𝜋}_{𝜃}\Vert {𝜋}_{\text{ ref}}\right)$ is the KL divergence between the current policy and the reference policy
• $𝜀$ is the clipping range of importance sampling ratio
• $𝐺$ is the group size
• ${𝑜}_{𝑖}$ is the $𝑖$-th response
• $𝑞$ is the question
• ${𝑜}_{𝑖,<𝑡}$ is the sequence of tokens before position $𝑡$ in response $𝑖$
• ${𝜋}_{𝜃}\left({𝑜}_{𝑖,𝑡}|𝑞,{𝑜}_{𝑖,<𝑡}\right)$ is the probability of generating token ${𝑜}_{𝑖,𝑡}$ under current policy
• ${𝜋}_{{𝜃}_{\text{old}}}\left({𝑜}_{𝑖,𝑡}|𝑞,{𝑜}_{𝑖,<𝑡}\right)$ is the probability of generating token ${𝑜}_{𝑖,𝑡}$ under the old policy