agent-readings

Training Agents

DeepSeek-R1

DeepSeek-R1 shows that large-scale reinforcement learning with verifiable rewards can elicit long-chain reasoning, then packages the result into a stronger cold-start-plus-RL model and distilled smaller models.

8Claims
8Evidence

Why It Matters

For agents, DeepSeek-R1 made verifiable reward reinforcement learning a concrete infrastructure pattern: if correctness can be checked, post-training can optimize trajectories at scale without hand-labeling every reasoning path.

Problem

  • Strong reasoning models relied heavily on supervised demonstrations, preference data, or closed training recipes.
  • Open post-training stacks lacked a clear frontier-scale example showing reasoning behavior emerging primarily from automatically checkable rewards.
  • Long chain-of-thought models introduced inference-time scaling, but the training recipe for producing them was not broadly visible.
  • Agent builders needed evidence for when to use verifiers and reward functions instead of treating reasoning as only prompt engineering.

Key gap: The paper asks whether an LLM can develop stronger reasoning through reinforcement learning against verifiable tasks, and how to turn the emergent but rough behavior into a usable model family.

Method

Use verifiable tasks as the reward source for RL, observe what reasoning behaviors emerge without supervised traces, then add cold-start data and staged post-training to improve readability, language control, general capability, and smaller-model transfer.

  • Train R1-Zero by applying GRPO to DeepSeek-V3-Base on reasoning tasks with rule-based accuracy and format rewards.
  • Track whether longer reasoning, self-reflection, verification, and strategy adaptation emerge during RL.
  • For R1, collect thousands of cold-start examples to make initial reasoning traces more readable and controlled.
  • Run reasoning-oriented RL from the cold-start model.
  • Use rejection sampling and supervised fine-tuning to broaden the model beyond narrow reasoning tasks.
  • Run a final RL stage for helpfulness, harmlessness, and all-scenario alignment.
  • Generate 800K training samples from DeepSeek-R1 and distill reasoning behavior into smaller dense Qwen and Llama models.

Claims

C1 · paper-supported

DeepSeek-R1-Zero demonstrates that reasoning behaviors can emerge from pure large-scale RL on a base model without supervised fine-tuning as a preliminary step.

Scope: Abstract, Introduction, Section 2.2, and Conclusion.

  • E-r1-zero-emergence: The paper describes R1-Zero as trained through large-scale RL without preliminary SFT and reports emergent reasoning behaviors such as self-reflection, verification, and dynamic strategy adaptation.

C2 · paper-supported

During R1-Zero training, AIME 2024 pass@1 rises from 15.6% to 71.0%, and majority voting reaches 86.7%.

Scope: Introduction and Section 2.2.4.

  • E-aime-r1-zero: The paper reports that R1-Zero's AIME 2024 pass@1 increases from 15.6% to 71.0%, and that majority voting improves the score to 86.7%.

C3 · paper-supported

DeepSeek-R1 adds cold-start data and a multi-stage pipeline to fix R1-Zero's practical weaknesses while preserving reasoning RL as the load-bearing mechanism.

Scope: Abstract, Introduction, Section 2.3, and Conclusion.

  • E-r1-pipeline: DeepSeek-R1 starts from thousands of cold-start examples, then uses reasoning-oriented RL, rejection sampling plus SFT, and a final RL stage for broader helpfulness and harmlessness.

C4 · paper-supported

The paper reports DeepSeek-R1 performance comparable to OpenAI-o1-1217 on reasoning tasks.

Scope: Abstract, evaluation summary, and Conclusion.

  • E-r1-o1-comparable: The authors summarize DeepSeek-R1 as achieving performance comparable to OpenAI-o1-1217 across a range of reasoning tasks.

C5 · paper-supported

Reasoning behavior transfers to smaller dense models through distillation from DeepSeek-R1 outputs.

Scope: Section 2.4, Section 3.2, and Conclusion.

  • E-distillation: The paper says DeepSeek-R1 generates 800K samples for student fine-tuning, and reports that DeepSeek-R1-Distill-Qwen-1.5B reaches 28.9% on AIME and 83.9% on MATH, outperforming GPT-4o and Claude-3.5-Sonnet on those math benchmarks.

C6 · paper-supported

The project releases DeepSeek-R1-Zero, DeepSeek-R1, and six distilled dense models spanning 1.5B to 70B parameters.

Scope: Abstract and project repository.

  • E-release: The paper and repository state that DeepSeek-R1-Zero, DeepSeek-R1, and six dense distilled models based on Qwen and Llama checkpoints are released, covering 1.5B, 7B, 8B, 14B, 32B, and 70B sizes.

C7 · paper-supported

The authors state that DeepSeek-R1 still falls short on function calling, multi-turn interaction, complex role-play, JSON output, language mixing outside Chinese and English, prompt sensitivity, and software-engineering RL efficiency.

Scope: Limitations and Future Work.

  • E-limitations: The paper lists weaknesses in function calling, multi-turn tasks, complex role-play, JSON output, non-Chinese/non-English language mixing, few-shot prompt sensitivity, and software-engineering RL because long evaluations slow the RL loop.

C8 · paper-supported

The paper reports unsuccessful attempts with process reward models, Monte Carlo tree search, and related search-style methods in their setting.

Scope: Discussion, Section 4.2.

  • E-unsuccessful-attempts: The discussion identifies process reward models, Monte Carlo tree search, and related search attempts as unsuccessful or insufficient in the authors' experiments.

Agent Infrastructure Takeaways

  • DeepSeek-R1 turns verifiers into training infrastructure: the key artifact is not a prompt but a reliable reward path for answer checking.
  • R1-Zero is the clean scientific result; R1 is the productizable recipe. Keep those separate when reasoning about what pure RL achieved.
  • For agents, RLVR works best where the environment can emit a crisp correctness signal, such as math, code, format, or executable tests.
  • Long reasoning traces are a serving concern. Better task success may come with higher token budgets, latency, and scheduler pressure.
  • Distillation turns expensive reasoning models into cheaper student policies, but the quality of the generated trajectories becomes part of the training data contract.
  • Function calling, JSON output, multi-turn interaction, and software engineering remain weak spots in the paper's own limitation section, so agent builders still need runtime checks.
  • Prompt sensitivity matters operationally: the paper recommends zero-shot, direct problem statements because few-shot prompting can degrade performance.

Caveats

  • The current arXiv record is v2 revised in January 2026 and also lists a Nature 2025 journal reference; this review treats the arXiv and Nature metadata as the current source record.
  • R1-Zero's pure-RL result should not be confused with DeepSeek-R1's final multi-stage recipe.
  • Benchmark parity with OpenAI-o1-1217 is scoped to reasoning tasks reported by the paper.
  • The evidence is strongest for verifiable reasoning tasks, not open-ended autonomous agents with side effects.
  • The authors explicitly report limitations in function calling, multi-turn interaction, JSON output, language mixing, prompt sensitivity, and software-engineering RL.
  • The paper's exact training data and infrastructure details are not fully reproducible from the public text alone.