Advantages and baselines

An advantage measures how much better a sampled action or rollout turned out than the policy should have expected in the same situation. It is the weight a policy-gradient update multiplies into each sampled token's log-probability, so its sign sets which agent behavior gets repeated and its magnitude sets how hard each rollout pushes.

$$A = R - b(s)$$

The baseline $b(s)$ is the expectation the reward is compared against.

Why rewards need a reference point

Suppose an agent solves 90% of the tasks in a batch and every success earns reward 1. With raw rewards as weights, every successful rollout is pushed up with the same strength, whether the task was trivial or the success was a lucky outlier, and every failed rollout receives zero weight.

Subtracting a baseline sets the reference point. A success on a task the policy almost always solves is expected, so its advantage is small. A success on a task it usually fails carries a large positive advantage, and a failure on an easy task a large negative one.

Baselines and the expected gradient

A baseline that depends on the state but not on the sampled action leaves the expected policy gradient unchanged:

$$\mathbb{E}_{a\sim\pi_\theta}\!\left[b(s)\,\nabla_\theta\log\pi_\theta(a\mid s)\right] = b(s)\sum_a \nabla_\theta\pi_\theta(a\mid s) = b(s)\,\nabla_\theta 1 = 0.$$

Any such $b(s)$ is allowed, and what it changes is the variance of the estimate. The conventional choice is the state value $V^\pi(s)$, the expected return from $s$ under the current policy. With that baseline the advantage estimates $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$, how much taking action $a$ changes the expected outcome relative to the policy's average from $s$. Centering each state's returns this way removes the variation that comes from some tasks being easier than others. $V^\pi$ is not the variance-minimizing baseline in general, since the optimum weights each return by the squared norm of its score function, but $V^\pi$ can be estimated from returns alone.

The derivation also sets a constraint. If the baseline uses the sampled action's own reward, the zero-mean argument breaks and the estimator becomes biased, which group baselines below have to handle.

Critics and GAE

A value model (or critic) is a learned network that predicts $V^\pi(s)$ from the current context. PPO in its classic RLHF form trains one alongside the policy, initialized from a similar LLM and fit by regression to observed returns. With a critic, the per-step advantage can be built from temporal-difference errors:

$$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t), \qquad \hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l\ge 0} (\gamma\lambda)^l\,\delta_{t+l}.$$

Generalized advantage estimation interpolates between two estimators with $\lambda$. At $\lambda = 0$ the advantage is the one-step error $\delta_t$, low variance but only as accurate as the critic. At $\lambda = 1$ the sum telescopes to the Monte Carlo return minus $V(s_t)$, unbiased but noisy. The PPO paper's reference settings use $\lambda = 0.95$.

A critic costs a second model of similar size, a second optimization problem, and a new failure mode: a poorly fit critic produces confident, wrong advantages. In a long agent rollout the context holds tool outputs from a sandbox whose state the critic cannot inspect, which makes predicting success from a prefix hard. What per-turn values buy when they are accurate is the subject of credit assignment.

Undiscounted returns

Classic RL discounts future reward by $\gamma < 1$ because episodes can be long or unbounded. LLM and agent tasks are finite: a rollout ends when the model stops, the harness hits a turn or token limit, or the task terminates, and most tasks give one reward at the end. Discounting per token would erase that reward. A reward 10,000 tokens away, discounted with $\gamma = 0.99$ per token, receives weight $0.99^{10{,}000}\approx 2.25\times10^{-44}$, so LLM RL almost always sets $\gamma = 1$.

With one terminal reward and $\gamma = 1$, every sampled token in rollout $i$ shares the return $R_i$. A rollout-level baseline $b_i$ gives all of them the same advantage $A_i = R_i - b_i$, while a critic can give each position its own.

Baselines from a group of rollouts

Because the return is one number per rollout, the baseline only needs the policy's expected reward on that task. Sampling several rollouts of the same task estimates it without training anything.

Leave-one-out (RLOO). For $n$ rollouts with rewards $r_1,\ldots,r_n$, each rollout's baseline is the mean of the others:

$$A_i = r_i - \frac{1}{n-1}\sum_{j\ne i} r_j.$$

The baseline excludes $r_i$, so it does not depend on rollout $i$'s actions and the estimator stays unbiased. Ahmadian et al. found that REINFORCE with this baseline outperforms PPO for RLHF at much lower cost.

Group mean. Subtracting the mean of all $n$ rewards, including $r_i$, is the baseline GRPO uses.

Worked example. Four rollouts of one task earn rewards $1, 0, 0, 1$.

Every RLOO value is $\tfrac{4}{3}$ times the group-mean value. The ratio holds for any group:

$$r_i - \bar r = \frac{n-1}{n}\left(r_i - \frac{1}{n-1}\sum_{j\ne i} r_j\right).$$

Mean-centering is RLOO scaled by $(n-1)/n$, so its expected gradient points in the same direction and is shorter by that factor. If all four rewards had been 1, both estimators would give zero everywhere, which is why difficulty filtering matters for group baselines.

Running baselines. A baseline can also be an exponential moving average of past rewards, per task or per agent role. It works with one rollout per prompt and suits multi-agent games, where a group mean over both sides of a zero-sum game is near zero whatever the policy does (self-play). The average trails a policy that is improving, so its advantages run slightly high during learning.

Baselines in agent RL

Group baselines dominate LLM RL with verifiable rewards. Generation is already the expensive part of a step, and $n$ rollouts per task add generation but no second model, while a critic adds a model that has to track the policy and still misjudges long tool-heavy prefixes. The price is resolution: a group baseline gives one advantage per rollout, and how that scalar is spread over turns is a separate choice.

A stronger baseline makes learning more selective but cannot repair the reward. If the scorer pays for a shortcut, the best shortcut in each group receives the largest advantage (reward hacking).