agent-readings · Tulu 3: Pushing Frontiers in Open Language Model Post-Training

Why It Matters

For agent training, Tulu 3 is the bridge from RLHF/DPO to verifiable-reward training: it shows how to use exact checks for math and instruction-following, while documenting the operational costs and failure modes of doing that at scale.

Problem

Open post-training recipes lagged proprietary labs in transparency, scale, and model quality.
Instruction tuning, preference tuning, evaluation, decontamination, and RL were often reported as fragments rather than a reproducible stack.
Open models could overfit popular benchmarks or lack clear unseen evaluations for core capabilities.
Verifiable-reward training was promising but under-documented as a practical post-training stage.

Key gap: The paper asks how to build a fully open, competitive post-training pipeline with released data, code, checkpoints, evaluation tooling, and recipe decisions.

Method

Build a staged open recipe: curate and decontaminate data around core skills, train SFT checkpoints, generate on-policy preference data and tune with length-normalized DPO, then run PPO with binary verifiable rewards for math and precise instruction-following before validating on development and unseen evaluations.

Identify core skills: knowledge recall, reasoning, math, coding, chat, safety, and precise instruction following.
Build development and unseen evaluation suites before finalizing recipes.
Curate public and synthetic instruction data and aggressively decontaminate against evaluations.
Train SFT checkpoints on Llama 3.1 base models.
Create on-policy preference data and train DPO checkpoints, selecting length-normalized DPO after ablations.
Train final 8B and 70B models with RLVR using GSM8K, MATH, and IFEval-style verifiers.
Scale the recipe to a 405B model with adjusted SFT/DPO/RLVR details.
Release weights, intermediate checkpoints, datasets, training code, evaluation code, and decontamination tooling.

Claims

C1 · paper-supported

Tulu 3 releases a complete open stack: model checkpoints, intermediate stages, datasets, training code, evaluation code, decontamination tooling, and a demo.

Scope: Abstract, Table 1, and release repositories.

E-release: The release lists base, SFT, DPO, reward-model, and final checkpoints; training code in open-instruct; evaluation code in olmes; decontamination tooling; preference-data inference; SFT, preference, and RLVR datasets; and a demo.

C2 · paper-supported

The Tulu 3 recipe is multi-stage: SFT, preference tuning with length-normalized DPO, then RLVR for selected verifiable skills.

Scope: Introduction, Sections 4-6.

E-recipe: The recipe starts from Llama 3.1 base models, curates/decontaminates data, trains SFT, runs preference tuning over on-policy data, then applies RLVR for verifiable math and instruction-following tasks.
E-dpo-ablation: Across explored preference-tuning algorithms on an early Tulu 3 checkpoint and UltraFeedback, only length-normalized DPO outperformed the base checkpoint overall; the best listed average score is 57.3 versus 55.7 SFT base.
E-rlvr: RLVR samples completions, applies a deterministic verifier, gives reward alpha when correct and zero otherwise, and optimizes the policy with PPO plus KL regularization.

C3 · paper-supported

On the paper's main evaluation suite, Tulu 3 8B and 70B outperform comparable open-weight instruct models at the same size and the 70B model matches or exceeds several smaller closed models on average.

Scope: Table 2.

E-main-results: The main suite reports averages of 65.1 for Tulu 3 8B and 76.2 for Tulu 3 70B, compared with 62.9 for Llama 3.1 8B Instruct, 74.1 for Llama 3.1 70B Instruct, 69 for GPT-4o mini, 70.6 for Claude 3.5 Haiku, and 75.3 for the best closed-model column in the table.

C4 · paper-supported

RLVR provides deterministic binary rewards for math and precise instruction-following, improving targeted test performance but not guaranteeing average performance gains across all evaluations.

Scope: Section 6.2.1 and Figure 19.

E-rlvr-targeted: RLVR improves targeted test performance for GSM8K, MATH, and prompts with constraints, and improves train-set verifiable rewards, but the paper states an increase in average score across all evaluations is not guaranteed.

C5 · paper-supported

Scaling post-training exposed concrete systems issues: loss aggregation semantics, DPO scaling cost, asynchronous RLVR inference/training, weight synchronization, and hardware restarts.

Scope: Sections 4.3, 5.4, 6.3, and 8.1.

E-sft-infra: The paper identifies mean-loss behavior under gradient accumulation and distributed training as changing sample/token weighting, and chooses sum loss with a 5e-6 learning rate and 2 epochs for SFT after experiments.
E-dpo-infra: The 8B DPO model trains for about 10 hours on 8 H100s, while the 70B DPO model trains for about 19 hours on 64 interconnected H100s with maximum sequence length 2048.
E-rlvr-infra: RLVR uses separate GPUs for inference and training, concurrent rollout/training to reduce idle time, and weight synchronization to the inference engine, while noting stale-data and reproducibility risks.
E-405b-infra: For the 405B RLVR setup, inference uses vLLM with 16-way tensor parallelism while the remaining 240 GPUs train; inference takes about 550 seconds, weight transfer about 25 seconds, and training about 1500 seconds per iteration.

C6 · paper-supported

The paper explicitly separates development and unseen evaluations and finds that some choices generalize while some instruction-following and knowledge decisions overfit development benchmarks.

Scope: Section 7.4 and Tables 31-33.

E-unseen-evals: The paper builds unseen evaluations aligned to core skills, reports that final checkpoints get the best average development and unseen performance, but notes overfitting in precise instruction-following and some knowledge/reasoning choices.

C7 · paper-supported

Several plausible training methods did not reliably improve performance, including online DPO in their setup and rejection sampling at the explored compute budget.

Scope: Section 8.2.

E-unfruitful: The paper reports that online DPO gave little or no improvement on GSM8K and degraded MATH in their setup, and that rejection sampling produced minimal gains for the compute required.

Agent Infrastructure Takeaways

Open recipes need artifacts, not just weights. The reusable contribution is data, code, evals, decontamination, intermediate checkpoints, and failed-method notes.
Verifiable rewards are the agent-training hinge: when a task has an executable checker, reward generation moves from human-time to compute-time.
Evaluation should be part of the training loop from day one, with development tasks separated from unseen tasks.
Post-training is a distributed systems problem at scale: rollout engines, learner GPUs, weight sync, stale data, hardware failures, and loss semantics all affect the model.
Synthetic data is useful only with aggressive curation and contamination checks; otherwise the apparent skill may just be benchmark leakage.
Negative results are recipe data. Knowing which obvious methods failed prevents cargo-cult post-training.

Caveats

The detailed PDF source used here is a v3 mirror, while the current arXiv entry is v5; the arXiv page confirms later 405B results and analyses.
The recipe is complex and resource-intensive, so reproducibility still requires substantial GPU infrastructure.
RLVR only applies cleanly where answer extraction and deterministic verification are reliable.
The paper focuses mostly on English data and evaluations, and lists long-context, multi-turn, and multilinguality as future-work areas.
Some comparisons with closed models rely on non-identical external reports, interpolated scores, or model-card numbers.
Unseen evaluations are unseen for Tulu 3, but the authors cannot verify that comparison models did not train on them.