Review date: 2026-05-12
Review author: Zhongzhu Zhou
Paper reviewed: DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Paper authors: Qiying Yu et al.
arXiv: 2503.14476v2, 2025-05-20
Venue/status: arXiv preprint
Source used for this review: src/related-documents/papers/2503.14476-DAPO.pdf
Short answer
DAPO is a paper about a very practical problem: how do we make large-scale reinforcement learning for reasoning LLMs actually reproducible? After OpenAI o1 and DeepSeek-R1, the community knew that reinforcement learning could make base models produce longer, more reflective chains of thought. But many important details remained hidden. A naive implementation of GRPO can move a model, but it often fails to match the behavior and accuracy of the strongest reasoning models.
The DAPO paper starts from that gap. The authors run reinforcement learning on a Qwen2.5-32B base model and find that a straightforward GRPO baseline reaches only 30 points on AIME 2024, far below the reported 47 for DeepSeek-R1-Zero-Qwen-32B. They then identify several training pathologies: entropy collapse, batches full of zero-gradient prompts, unhealthy response-length growth, and noisy rewards for truncated long responses. DAPO, short for Decoupled Clip and Dynamic sAmpling Policy Optimization, is their recipe for fixing those problems.
The headline result is easy to remember. DAPO trains Qwen2.5-32B to 50 avg@32 on AIME 2024, beating the DeepSeek-R1-Zero-Qwen-32B number cited in the paper while using roughly half as many training steps. The contribution is not just a single formula. It is a set of four techniques that work together:
- Clip-Higher: decouple the lower and upper PPO/GRPO clipping bounds, then raise the upper bound to reduce entropy collapse.
- Dynamic Sampling: keep only prompts whose sampled responses include both successes and failures, so the group-relative advantage is informative.
- Token-Level Policy Gradient Loss: normalize loss by tokens rather than by samples, which matters when reasoning traces have very different lengths.
- Overlong Reward Shaping: replace a crude penalty for truncated responses with softer length-aware punishment, reducing reward noise.
My main takeaway is that DAPO is not merely "GRPO with better hyperparameters." It is closer to a systems paper for RL post-training. The method says: if you want long-chain-of-thought RL to work, you must control exploration, gradient usefulness, credit weighting across tokens, reward noise, data format, and monitoring metrics together. Each part looks modest in isolation, but Table 1 shows a progressive path from 30 to 50 AIME points.
1. Prerequisites
1.1 Inference scaling and long-chain reasoning
Modern reasoning models often improve by spending more computation at inference time. Instead of answering immediately, the model generates a longer internal reasoning trajectory, checks intermediate steps, backtracks, and only then gives an answer. This is usually called test-time scaling or inference scaling. The important intuition is simple:
1 | short answer path: |
For math and programming tasks, the second path can be much stronger. A model may not know the final answer immediately, but it may be able to discover it by writing intermediate equations, trying a construction, noticing a contradiction, and correcting itself.
Reinforcement learning is attractive here because many reasoning tasks have a verifiable final answer. If the final answer is correct, give a positive reward. If it is wrong, give a negative reward. Over many rollouts, the model should learn to produce trajectories that tend to lead to correct answers. The difficulty is that the reward arrives only at the end. The model may write thousands of tokens before the verifier says correct or incorrect. That makes optimization noisy and sensitive to small implementation choices.
1.2 PPO, clipping, and why the clip matters
PPO, or Proximal Policy Optimization, is a policy-gradient algorithm designed to avoid updates that are too large. The model samples an action from an old policy, then the new policy is trained to make good actions more likely and bad actions less likely. PPO compares the new probability to the old probability through an importance ratio:
1 | ratio = probability under new policy / probability under old policy |
If the ratio changes too much, PPO clips it. With a default clip such as 0.2, the ratio is restricted roughly to the interval [0.8, 1.2]. This is a stability mechanism. It prevents the policy from making extreme jumps after seeing one noisy batch.
For language models, each generated token is an action. During reasoning RL, a "good" response is not one action but a long sequence of token choices. Some tokens are common and already likely under the base model. Other tokens are rare exploration moves: "wait," "let me check," a different algebraic substitution, or a proof branch that the base model would not normally choose. DAPO argues that the upper clip can make it hard to increase the probability of those low-probability but useful exploratory tokens.
This is the starting point for Clip-Higher. The paper keeps the lower clipping bound conservative but raises the upper clipping bound, allowing positive-advantage tokens more room to become likely.
1.3 GRPO and group-relative advantages
GRPO, or Group Relative Policy Optimization, removes the value model used in PPO. Instead of estimating a value function, GRPO samples a group of outputs for the same prompt and normalizes their rewards inside the group.
Suppose we ask one math question and sample 16 responses. Some are correct and some are wrong. The correct responses get higher relative advantages; the wrong responses get lower relative advantages. This gives a training signal without a learned critic.
1 | prompt q |
This is elegant, but it has an important failure case. If all 16 responses are correct, every reward is the same and the normalized advantages become zero. If all 16 responses are wrong, the same thing happens. Such prompts consume rollout budget but contribute little useful gradient. DAPO's dynamic sampling is built around this observation.
1.4 Verifiable rewards and reward noise
DAPO focuses on math problems where the final answer can be checked by a rule. The reward is essentially:
1 | +1 if predicted answer is equivalent to the ground truth |
This avoids the need for a learned reward model and reduces reward hacking risk. However, a rule-based reward is only as clean as the answer extraction and the task format. If the final answer appears in a complicated symbolic form, the parser can be wrong. If a response is truncated because it reached the maximum length, the answer may be missing even though the reasoning was promising. If all truncated responses are punished harshly, the policy may receive a misleading signal.
The paper therefore pays attention to two less glamorous but important details:
- transform dataset answers into easy-to-parse integer forms;
- shape the reward for overlong responses more carefully.
This is one of the reasons DAPO feels like a real training recipe rather than only an objective function.
1.5 Why long-CoT RL is a systems problem
Long-chain-of-thought reinforcement learning touches many subsystems at once:
- rollout generation must sample many long responses;
- the verifier must parse answers reliably;
- the optimizer must handle high-variance sparse rewards;
- sequence length must be monitored because longer is not always better;
- entropy must be monitored because both collapse and uncontrolled randomness are bad;
- training throughput depends on the longest samples in a synchronized batch.
Changing one component often affects the rest. For example, increasing the maximum generation length may produce richer reasoning, but it also increases latency and creates more truncated samples. Raising entropy may improve exploration, but too much entropy can produce repetition or gibberish. Filtering out easy prompts may improve gradient quality, but it changes the distribution of training examples.
DAPO's practical value is that it treats these interactions explicitly. It asks not only "which RL objective is correct?" but also "which batch actually gives a useful gradient, which responses should be weighted, and which monitoring curves tell us the run is healthy?"
2. What this paper does
The paper proposes a reproducible large-scale RL system for training reasoning LLMs. It starts from a baseline that many practitioners would naturally try: GRPO with rule-based math rewards. That baseline works only partially. It improves the model, but on AIME 2024 it reaches 30 avg@32, which is not enough to match the strongest public reasoning results.
DAPO is then introduced as a sequence of fixes. The conceptual flow is:
1 | Qwen2.5-32B base model |
The paper also releases code built on the verl framework and a curated math dataset called DAPO-Math-17K. That matters because the community's difficulty was not only conceptual. Many groups could read the high-level DeepSeek-R1 report but still fail to reproduce comparable results. DAPO tries to make the hidden engineering recipe visible.
The authors organize the method around four named techniques. I like this structure because each technique maps to a concrete symptom in training:
| Symptom | Why it hurts | DAPO component |
|---|---|---|
| Entropy drops too fast | model stops exploring useful reasoning paths | Clip-Higher |
| Many prompt groups have all correct or all wrong samples | group-relative advantage becomes zero | Dynamic Sampling |
| Very long responses are averaged like short responses | token-level learning signal is distorted | Token-Level Policy Gradient Loss |
| Truncated outputs are punished too bluntly | correct-looking partial reasoning can be mislabeled | Overlong Reward Shaping |
The central message is that reasoning RL is not bottlenecked by one magic algorithm. It is bottlenecked by a collection of small mismatches between the objective and the actual behavior of long generated sequences.
3. Method details
3.1 Starting from GRPO, then removing unnecessary KL pressure
The paper reviews PPO and GRPO before defining DAPO. In PPO, a value function estimates advantages. In GRPO, the value function is removed and the advantage is computed by comparing responses inside the same prompt group:
1 | advantage(response i) = (reward_i - group_reward_mean) / group_reward_std |
This is a natural fit for verifiable math RL. For each prompt, sample multiple responses. Correct responses should move up; incorrect responses should move down. No critic network is needed.
The paper also removes the KL penalty to the reference policy. In classical RLHF, the KL term keeps the aligned model from drifting too far from the initial supervised model. That is useful when the goal is to preserve chat behavior while nudging preferences. But in long-CoT reasoning, the goal is partly to create a new distribution: longer reasoning, backtracking, self-verification, and problem-specific exploration. A strong KL penalty can prevent that shift. DAPO therefore focuses on clipped policy updates without a direct reference-model KL term.
This design choice is important. It tells us that the authors view reasoning RL less like mild alignment tuning and more like behavior discovery under a verifiable reward. The model must be allowed to move away from the base distribution, but not so violently that training collapses.
3.2 Clip-Higher: raise the upper clip to protect exploration
Figure 2 is the first major evidence in the paper. It compares training with and without Clip-Higher. Without Clip-Higher, the entropy of the actor model's generated probabilities falls rapidly. The model becomes too deterministic too early. Accuracy also lags. With Clip-Higher, entropy stays healthier and AIME accuracy improves.
The mechanism is subtle but intuitive. PPO-style clipping has two sides:
- the lower side limits how much the probability of a token can be reduced;
- the upper side limits how much the probability of a token can be increased.
If a token already has probability 0.9, multiplying by 1.2 gives 1.08, which saturates anyway. The upper clip does not meaningfully stop that high-probability token from becoming dominant. But if a useful exploratory token has probability 0.01, multiplying by 1.2 only raises it to 0.012. That is a very small increase. The upper clip therefore restricts low-probability exploration tokens more than it restricts already-common exploitation tokens.
DAPO decouples the two clip bounds:
1 | lower bound: 1 - epsilon_low |
In the experiments, the authors use epsilon_low = 0.2 and epsilon_high = 0.28. They raise only the upper bound. Keeping the lower bound conservative avoids aggressively suppressing low-probability tokens to zero, while raising the upper bound gives positive-advantage exploratory tokens more room to grow.
Figure 3 supports the diagnosis. The mean probability of up-clipped tokens is low, often below 0.2, indicating that the tokens being clipped on the upward side are not already-obvious high-probability tokens. Figure 3 also shows the increasing ratio of prompts with accuracy equal to 1, which motivates the next component, dynamic sampling.
My interpretation is that Clip-Higher is a targeted fix for the asymmetry between exploration and exploitation in token space. The default PPO clip was designed as a generic trust-region heuristic. In long-CoT RL, the useful distribution shift may require making initially rare reasoning moves less rare. A slightly higher upper clip is a simple way to let that happen without abandoning clipping entirely.
3.3 Dynamic Sampling: spend gradient budget on informative prompt groups
GRPO's group-relative advantage needs variation inside the group. If every sampled response for a prompt is correct, the normalized rewards are all equal. If every sampled response is wrong, they are again all equal. In both cases, the advantage is zero or nearly useless. The prompt has consumed rollout budget, but it does not tell the optimizer which behavior to reinforce.
DAPO defines the useful condition as:
1 | 0 < number of correct responses in the group < G |
where G is the number of sampled responses for one prompt. In plain language: keep the prompt only if the current model sometimes solves it and sometimes fails. Those are the examples where the batch contains contrastive evidence.
The dynamic sampling procedure over-samples prompts and fills a buffer only with informative groups. If the buffer is not full, sampling continues. This makes the actual sampling cost dynamic. A stronger model will produce more all-correct groups on easy prompts, so the sampler must search harder for partially solved prompts.
At first glance this sounds expensive. But the paper argues that in synchronized RL systems, generation time is often dominated by long-tail samples. Filtering uninformative groups does not necessarily increase wall-clock time much, and Figure 6 shows that dynamic sampling can reach the same or better performance with fewer training steps.
The deeper point is about gradient quality. In supervised learning, every labeled example usually contributes a loss. In group-relative RL, not every prompt group contributes useful contrast. DAPO explicitly separates rollout quantity from gradient usefulness.
3.4 Token-Level Policy Gradient Loss: do not let long reasoning traces vanish
The original GRPO loss is described as sample-level. It first averages token losses within each response, then averages across responses. That means a short response and a very long response can contribute equal total weight at the sample level.
For long-CoT reasoning, this can be distorted. A long response may contain many meaningful reasoning moves. If it is high quality, averaging the whole sequence into one sample-level loss underweights all those useful tokens. If it is low quality, full of repetition or gibberish, sample-level averaging also underweights the many bad tokens that should be suppressed.
DAPO instead uses token-level loss aggregation. The paper's objective normalizes across tokens, so tokens in longer responses are not hidden inside a per-sample average. From an optimization perspective, this makes the unit of learning closer to the actual unit of generation: the token.
Figure 4 illustrates why this matters. Without token-level loss, entropy and response length can grow in an unhealthy way. With token-level loss, the model has a more direct signal for the patterns inside long responses. The paper notes that token-level loss adds less headline accuracy than some other components, but improves training stability and length behavior.
This is a useful practical lesson. A change can be important even if it does not produce the largest single ablation jump. For production training, stability and predictable length dynamics often matter as much as the final leaderboard number.
3.5 Overlong Reward Shaping: avoid confusing truncation with bad reasoning
Long reasoning models need a maximum generation length. In the DAPO experiments, the expected maximum length is 16,384 tokens, with an additional 4,096-token soft punishment cache, for a total generation limit of 20,480 tokens. This is a large budget, but some samples can still hit the limit.
The naive handling is to assign a punitive reward to truncated samples. The problem is that truncation is not always the same as incorrect reasoning. A response may be exploring a valid solution path but fail to finish within the limit. If the system simply marks it as wrong, the model receives noisy feedback: "this reasoning pattern is bad," when the real problem was length management.
The authors first test Overlong Filtering, which masks the loss of truncated samples. Figure 5 shows that this stabilizes training and improves AIME performance compared with blunt punishment. Then they propose Soft Overlong Punishment, a length-aware penalty. The penalty is zero before the soft interval, decreases linearly inside the interval, and becomes -1 after the maximum length.
In simplified form:
1 | if length <= Lmax - Lcache: |
This gives the model a smoother signal: being slightly too long is discouraged, but not treated the same as a fully failed answer. For long-CoT RL, this is exactly the kind of reward shaping that can decide whether training remains stable.
3.6 Dataset transformation: make rewards parseable
DAPO uses a curated dataset called DAPO-Math-17K. The paper says the data comes from web sources and official competition homepages, with web scraping and manual annotation. A key transformation is that answers are converted into integer forms when possible.
This may sound like a data-cleaning footnote, but it is central to rule-based rewards. Math answers can be expressions, fractions, radicals, tuples, or textual explanations. A reward function that must decide whether 11 - 2 sqrt(6) is equivalent to another expression can make mistakes. If the problem is rewritten so the expected answer is an integer such as 19, answer checking becomes more reliable.
Appendix A gives an example. The original problem asks for the smallest possible value of x, whose answer is 11 - 2 sqrt(6). The transformed problem asks for k + m + n when the answer is written as k - m sqrt(n), making the final answer 19. The paper uses LLM-assisted transformation with structured reasoning steps: extract answer format, rewrite the problem, solve the modified problem, and provide the integer answer.
For me, this is one of the most reproducibility-relevant parts of the paper. Many RL failures are not caused by the policy optimizer alone. They come from messy reward interfaces. DAPO improves that interface.
3.7 Algorithm 1 as the full training loop
Algorithm 1 presents DAPO as a loop:
1 | for each training step: |
The algorithm is short, but each line hides a practical concern. Sampling many long responses is expensive. Computing rewards requires robust answer parsing. Dynamic sampling changes the batch composition. Updating the policy requires careful normalization. The paper's contribution is to make those concerns explicit enough to reproduce.
4. Experiment setup
4.1 Model and training framework
The main model is Qwen2.5-32B base. This choice is important because it matches the scale of DeepSeek-R1-Zero-Qwen-32B, making the headline comparison more meaningful. The training framework is verl, an open-source RLHF/RL training system.
The baseline algorithm is naive GRPO with group reward normalization. DAPO then progressively adds the four techniques described above.
4.2 Data and reward
The training data is DAPO-Math-17K, a curated set of 17K math prompts paired with integer answers. The reward is rule-based final-answer correctness:
1 | reward = +1 if predicted answer is equivalent to ground truth |
The paper focuses on mathematical reasoning. The authors say the approach can transfer to other tasks, but the reported main experiments are math-centered.
4.3 Rollout and optimization hyperparameters
The paper gives several concrete hyperparameters:
- optimizer: AdamW;
- learning rate: constant
1e-6; - warm-up: linear warm-up over 20 rollout steps;
- prompt batch size: 512;
- responses per prompt: 16;
- training mini-batch size: 512;
- gradient updates per rollout step: 16;
- expected maximum length: 16,384 tokens;
- soft punishment cache: 4,096 tokens;
- maximum generation length: 20,480 tokens;
- clipping:
epsilon_low = 0.2,epsilon_high = 0.28; - evaluation temperature: 1.0;
- evaluation top-p: 0.7.
These details matter because long-CoT RL is sensitive to them. A paper that only says "we use GRPO" is not enough to reproduce the system. DAPO is valuable partly because it reports the recipe in enough detail to guide implementation.
4.4 Evaluation metric
The main evaluation is AIME 2024 avg@32. The authors repeat the evaluation set 32 times and report average accuracy, which reduces noise from sampling. This is a good fit for stochastic reasoning models: a model may solve a problem in some rollouts and fail in others.
AIME is a demanding math benchmark, but it is still a narrow slice of reasoning. It measures competition-style math, not coding, science QA, theorem proving, or open-ended agentic tasks. That limitation matters when interpreting the result.
5. Results and analysis
5.1 Figure 1: the headline AIME result
Figure 1 shows DAPO's AIME 2024 score over training. The headline claim is that DAPO reaches 50 avg@32 with Qwen2.5-32B, outperforming the paper's cited DeepSeek-R1-Zero-Qwen-32B score of 47. The authors also emphasize that DAPO uses about 50% of the training steps required by that comparison.
This figure is important because it frames DAPO as more than an ablation improvement. The paper claims that a fully open recipe can reproduce and slightly exceed a previously closed high-performing RL reasoning result at the same model scale.
The careful interpretation is: DAPO is convincing evidence that the gap between naive GRPO and strong reasoning RL is largely in the recipe. It does not prove that DAPO is universally superior to every possible R1-style implementation, because many details of proprietary or semi-open systems remain unknown. But it does show that the public community can reach this performance region with a transparent method.
5.2 Table 1: the progressive path from 30 to 50
Table 1 is the most useful table in the paper:
| Model / technique stack | AIME24 avg@32 |
|---|---|
| DeepSeek-R1-Zero-Qwen-32B | 47 |
| Naive GRPO | 30 |
| + Overlong Filtering | 36 |
| + Clip-Higher | 38 |
| + Soft Overlong Punishment | 41 |
| + Token-level Loss | 42 |
| + Dynamic Sampling (DAPO) | 50 |
The table tells a story in stages:
- Naive GRPO reaches 30, so the basic signal is real but incomplete.
- Overlong filtering adds 6 points, showing that truncated-response reward noise is a major issue.
- Clip-Higher adds 2 points, consistent with the entropy-collapse diagnosis.
- Soft overlong punishment adds 3 points, improving over masking by giving a smoother length signal.
- Token-level loss adds 1 point, but also improves stability and length dynamics.
- Dynamic sampling adds 8 points, the largest final jump, indicating that gradient usefulness is a dominant bottleneck.
The dynamic sampling gain is especially revealing. It suggests that once the model becomes moderately capable, many prompts become either too easy or too hard under group sampling. Training on them wastes budget. The most valuable prompts are those on the boundary where the model sometimes succeeds.
5.3 Figures 2 and 3: entropy collapse and clipping asymmetry
Figure 2 compares accuracy and generation entropy with and without Clip-Higher. The entropy curve is the more diagnostic one. Without Clip-Higher, entropy collapses quickly. With Clip-Higher, entropy remains higher and accuracy improves.
Figure 3a shows the mean probability of up-clipped tokens. Because those probabilities are low, the paper's explanation is plausible: the upper clip is mostly limiting low-probability exploratory tokens, not just restraining already-dominant tokens. Figure 3b shows that the ratio of prompts with all sampled outputs correct increases during training, which motivates dynamic sampling.
Together, these figures show two phases of RL difficulty:
1 | early training: |
That phase distinction is useful for practitioners. A single metric such as validation accuracy is not enough. You need entropy, response length, reward, and group accuracy distribution to understand what the run is doing.
5.4 Figure 4: token-level loss controls length and entropy
Figure 4 compares token-level loss with sample-level loss. Without token-level loss, the model can show unhealthy increases in entropy and response length. With token-level loss, training dynamics are more controlled.
This result fits the intuition that long responses should not be compressed into the same sample weight as short responses. If a 5,000-token response contains repeated low-quality reasoning, a sample-level average may hide the repeated pattern. Token-level aggregation makes every generated token more visible to the optimizer.
The broader lesson is that reduction semantics matter. In deep learning code, a line such as loss.mean() looks innocent. But in long-sequence RL, whether we average over tokens first or aggregate tokens globally changes the training signal.
5.5 Figure 5: overlong handling is not a side detail
Figure 5 compares training with and without overlong filtering. The version with filtering is more stable and reaches higher AIME accuracy. This result supports the claim that naive penalties for truncated samples inject reward noise.
The paper then prefers soft overlong punishment in the final stack. I read this as a compromise between two bad extremes:
- If we fully punish every truncated sample, we may penalize promising reasoning.
- If we fully ignore every truncated sample, the model may not learn to manage length.
The soft penalty says: length matters, but the penalty should be gradual.
5.6 Figure 6: dynamic sampling improves convergence
Figure 6 shows training progress before and after dynamic sampling on a baseline setting. The dynamic-sampling run achieves the target performance faster. This is one of the most practically important findings. Even if dynamic sampling requires extra rollout attempts, it can reduce the number of optimization steps needed because the retained batches carry stronger gradient information.
In other words, DAPO is not simply spending more compute. It is spending compute on examples with nonzero contrast.
5.7 Figure 7: what to monitor during long-CoT RL
Figure 7 plots response length, reward score, generation entropy, and mean probability. The paper emphasizes that these metrics are essential monitoring signals.
I would summarize the monitoring logic as follows:
- Length: should increase enough to allow richer reasoning, but uncontrolled length growth can indicate instability.
- Training reward: should generally improve, but high training reward may not correlate strongly with validation accuracy because of overfitting.
- Entropy: should not collapse too quickly, but excessively high entropy can mean gibberish or repetition.
- Mean probability: complements entropy by showing how sharp the generated distribution is.
This section is valuable because it acknowledges the messy nature of RL experimentation. A modification can look theoretically reasonable and still fail because it interacts badly with data, reward parsing, length limits, or sampling settings. Good monitoring is part of the method.
5.8 Tables 2 and 3: reflective behavior emerges
Table 2 shows an example where the model starts to reflect during problem solving: it pauses, rethinks the geometry, and revises its approach. Table 3 in the appendix gives another supplementary case with a combinatorial problem where the model notices an inconsistency and changes strategy.
These cases are anecdotal, not rigorous proof of reasoning emergence. But they are still useful. The goal of long-CoT RL is not just to raise final answer accuracy; it is to induce behaviors such as checking, backtracking, and correction. The examples show the kind of behavior DAPO is trying to reinforce.
A stronger future paper could quantify these behaviors more systematically. For example, it could measure the frequency of explicit self-checking phrases, backtracking markers, or solution-branch changes and correlate them with success.
6. Limitations and boundary conditions
6.1 The evidence is mostly math-centered
DAPO's main result is on AIME 2024 with math training data. That is a strong and relevant benchmark, but it does not automatically imply the same gains for coding, tool use, scientific reasoning, long-context QA, or agentic workflows. Rule-based rewards are easiest in math because final answers can often be checked. Other domains have messier correctness criteria.
The method should transfer best to tasks with:
- verifiable final outputs;
- many prompts where the model sometimes succeeds and sometimes fails;
- enough rollout budget to sample multiple responses per prompt;
- reliable answer extraction.
It may transfer less cleanly to tasks where correctness is subjective, partial-credit scoring is needed, or the final answer cannot be checked automatically.
6.2 The comparison to closed recipes is necessarily incomplete
The paper compares DAPO to DeepSeek-R1-Zero-Qwen-32B numbers, but the exact DeepSeek recipe is not fully public. That is the motivation for DAPO, but also a limitation of the comparison. We cannot know whether differences are due to objective details, data, infrastructure, filtering, hyperparameters, model checkpoints, or evaluation protocol.
The fair reading is: DAPO demonstrates an open recipe that reaches the same performance region. It should not be read as a final controlled comparison against every hidden detail of R1-style training.
6.3 Dynamic sampling changes the training distribution
Dynamic sampling filters out all-correct and all-wrong prompt groups. This improves gradient usefulness, but it also changes the effective distribution of prompts. The model trains more on boundary cases and less on easy or impossible cases.
That is usually good for optimization, but there are possible side effects:
- easy prompts may receive less reinforcement for concise correct behavior;
- very hard prompts may be underrepresented until the model improves;
- the curriculum depends on the current model's sampling behavior;
- evaluation domains with different difficulty distributions may respond differently.
In practice, this means dynamic sampling should be monitored carefully. It is a curriculum mechanism, not just a variance-reduction trick.
6.4 Length shaping is task-dependent
The chosen length limits, 16,384 plus a 4,096-token cache, fit the paper's math setting and model scale. Other tasks may require different budgets. A theorem-proving problem might need longer traces. A production chatbot may need much shorter answers. A code task may need room for both reasoning and code.
Soft overlong punishment is a useful idea, but the exact thresholds should not be copied blindly. They are part of the task definition.
6.5 Data transformation can introduce hidden bias
Transforming math answers into integers improves reward reliability, but it can also alter the task distribution. Some problems are easier to transform than others. The transformation process uses LLM assistance and manual annotation, which may introduce selection effects. The paper gives examples and releases data, which helps, but practitioners should still audit the transformed dataset before treating it as a neutral benchmark.
6.6 Compute and infrastructure details are still partly implicit
The paper gives many hyperparameters and releases code, but large-scale RL with a 32B model requires substantial GPU infrastructure. Important deployment questions remain:
- how many GPUs are needed for a practical reproduction;
- how rollout generation and training are parallelized;
- how failures and stragglers are handled;
- how much wall-clock time the full run takes;
- how sensitive the result is to hardware topology and inference engine choices.
The use of verl helps, but reproducing the headline number is still a serious systems project.
7. Reproducibility and practical notes
7.1 What DAPO makes reproducible
DAPO is unusually useful for reproduction because it releases three things together:
- algorithmic recipe — the four named techniques and the DAPO objective;
- training code — built on the open-source verl framework;
- dataset — DAPO-Math-17K with transformed answer formats.
This combination matters. An algorithm without data is hard to reproduce. Data without training code leaves too many implementation choices. Code without a clear paper recipe can be difficult to adapt. DAPO gives the community a more complete starting point.
7.2 A practitioner checklist
If I were implementing DAPO-style RL, I would not start by only copying the objective. I would build the following checklist:
1 | Data and reward |
This checklist is the practical spirit of the paper. DAPO is not one knob. It is a runbook.
7.3 When I would use DAPO
I would consider DAPO when the task has verifiable rewards and the model can generate diverse attempts. Examples include math, programming contest problems, theorem proving with a verifier, unit-test-based code generation, or structured data transformation tasks with exact checking.
I would be more cautious in open-ended writing, dialogue quality, safety preference learning, or multi-objective human preference tasks. Those settings often require learned reward models or human preference data, and the reward noise profile is different.
7.4 How this paper fits with PPO, GRPO, and RLVR
DAPO sits in the broader trend of reinforcement learning with verifiable rewards. PPO introduced stable clipped policy updates. GRPO simplified critic-free optimization by using group-relative rewards. DeepSeekMath and DeepSeek-R1 popularized GRPO-style reasoning RL. DAPO asks what extra recipe is needed to make this style work at scale.
The answer is not "throw away GRPO." The answer is:
1 | keep group-relative policy optimization, |
That is why the paper is worth reading. It translates a high-level RL idea into an operational recipe.
8. Final assessment
DAPO is a strong paper because it addresses the part of LLM reinforcement learning that practitioners actually struggle with: the gap between a public high-level method and a working large-scale run. Its contribution is not a single surprising theorem. It is a transparent engineering recipe backed by ablations, monitoring curves, code, and data.
The most important empirical evidence is Table 1. Moving from 30 to 50 AIME points requires several fixes. This tells us that naive GRPO is not enough, but also that the missing pieces are understandable. Entropy collapse can be mitigated. Zero-gradient prompt groups can be filtered. Token weighting can be corrected. Truncation noise can be softened. Dataset answers can be made parseable.
The main caution is scope. DAPO is most convincing for math-style RLVR. Its ideas are likely useful elsewhere, but the exact recipe should be revalidated for other domains, models, and reward types.
For researchers, DAPO is a reproducible baseline for studying reasoning RL. For system builders, it is a checklist of failure modes to monitor. For anyone trying to understand why R1-like training is hard, it gives a clear answer: the algorithm is only one layer. The training recipe, data interface, reward shaping, and systems feedback loops are just as important.
References
- Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2, 2025.
- Zhihong Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
- John Schulman et al. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017.
- Daya Guo et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948, 2025.
- Guangming Sheng et al. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv:2409.19256, 2024.
Review written on 2026-05-12.