GRPO (Group Relative Policy Optimization)

GRPO (Group Relative Policy Optimization) is a policy-gradient reinforcement learning algorithm for language models that samples a group of answers to the same prompt, scores each one, and uses the group's own average score as the baseline — so no separate value network (critic) has to be trained. It was introduced by DeepSeek researchers in the DeepSeekMath paper (Shao et al., February 2024), which describes it as "a variant of Proximal Policy Optimization (PPO)" that "foregoes the critic model, instead estimating the baseline from group scores." It later became the training algorithm for DeepSeek-R1 and, as of October 2026, is the default starting point for reinforcement fine-tuning of open-weight models.

How It Works

For each prompt, the current policy generates a group of G completions. A reward function or reward model scores each completion, and the rewards are normalised inside the group: subtract the group mean, divide by the group standard deviation. In the outcome-supervised form, that normalised number becomes the advantage for every token of the completion. The update itself is PPO's clipped objective — the ratio of new to old token probabilities, clipped to a small interval — plus a KL-divergence term that keeps the policy near a reference model. DeepSeekMath adds the KL term directly to the loss rather than folding it into the reward.

The original experiment sampled 64 outputs per question with a KL coefficient of 0.04 and reported that RL lifted the 7B instruct model from 82.9% to 88.2% on GSM8K and from 46.8% to 51.7% on MATH. DeepSeek-R1 (DeepSeek-AI, January 2025; later published in Nature) used the same algorithm with rule-based rewards for answer accuracy and output format, and reports the R1-Zero variant's average pass@1 on AIME 2024 rising from 15.6% to 77.9% over training.

GRPO Versus PPO and Preference Methods

Method	Training signal	Extra models needed	Data
GRPO	Scalar reward per sampled answer, compared within a group	None required; an optional frozen reference policy for the KL term	Online: the model generates its own rollouts
PPO (as used in RLHF)	Reward minus a learned per-token value estimate	A value model, typically comparable in size to the policy, and usually a learned reward model	Online rollouts
DPO	Pairs of preferred and rejected answers	A frozen reference policy	Offline, fixed preference dataset

The DeepSeekMath authors give two reasons for dropping the critic: a value model roughly the size of the policy is a large memory and compute burden, and in language tasks reward usually arrives only at the final token, which makes an accurate per-token value function hard to fit. Preference methods solve a different problem. DPO needs no sampling and no reward function, but it can only pull a model towards answers someone has already ranked; GRPO explores, and can therefore find behaviour that is in no dataset — provided something can score it. The two are sometimes combined: G-Zero (Huang et al., May 2026) trains a task proposer with GRPO and a generator with DPO.

Role in Open-Model Training in 2026

GRPO is cheap enough to run on small hardware, which is most of its appeal. Hugging Face's TRL library ships a GRPOTrainer; its documentation notes that the KL coefficient defaults to zero because later studies found the term non-essential, and exposes alternative loss types (dapo, dr_grpo, sapo). Unsloth's reinforcement-learning guide, read in October 2026, documents GRPO with LoRA and QLoRA adapters, states that 5 GB of VRAM is enough for models of 1.5B parameters or fewer, and advises at least 500 rows of data and at least 300 steps before expecting reward to move. Its advice on rewards is to write a rubric of several small verifiable checks rather than one all-or-nothing score.

The algorithm is also used well beyond maths problems. A May 2026 alphaXiv write-up reports training Qwen3.5-4B with GRPO (in the SkyRL framework, after a supervised cold start) to operate as a recursive language model, reaching an average rubric score of 0.59 against 0.607 for Claude Sonnet 4.6. The authors are explicit that this is a single task — evidence selection from scientific documents — with a long task-specific system prompt, so it shows feasibility rather than general parity.

Known Failure Modes

No signal when the group agrees. If every sample in a group earns the same reward, all advantages are zero and the prompt contributes no gradient. DAPO (Yu et al., March 2025) addresses this with dynamic sampling that filters out such groups.

Entropy collapse. The DAPO authors report policy entropy falling quickly under naive GRPO, and that their four changes (decoupled clipping, dynamic sampling, token-level loss, a penalty for over-long outputs) raised a Qwen2.5-32B run from 30 to 50 points on AIME 2024.

Length bias. Liu et al. (March 2025) show that averaging the loss over each response's length biases optimisation towards longer responses, particularly incorrect ones; their Dr. GRPO variant divides by a constant instead.

Reward dependence. GRPO optimises whatever the scorer measures. The R1 paper says its authors avoided neural reward models for reasoning tasks because they were "susceptible to reward hacking during large-scale reinforcement learning", and lists reward hacking as an open obstacle for tasks, such as writing, that lack a reliable verifier. Each update also costs G full rollouts per prompt, which is expensive for long agentic episodes in RL environments.