agent-readings

Training Agents

InstructGPT

InstructGPT made broad instruction following work by training GPT-3 with demonstrations, human preference rankings, a reward model, and PPO.

7Claims
7Evidence

Why It Matters

This is the baseline recipe for post-training agent models: collect task-facing behavior data, model preferences, optimize against that model, and measure the resulting assistant on user-distribution prompts instead of only academic NLP tasks.

Problem

  • GPT-3-style pretraining optimized next-token prediction, not following user instructions helpfully and safely.
  • Prompt engineering could elicit capabilities but did not reliably reduce unhelpful, untruthful, or toxic behavior.
  • Public NLP instruction datasets such as FLAN and T0 did not match deployed API prompt distributions.
  • Safety and helpfulness judgments are subjective enough that simple automatic metrics were insufficient.

Key gap: The paper asks how to adapt a general pretrained language model to follow broad real user instructions while improving helpfulness, truthfulness, and harmlessness.

Method

Start with GPT-3, train an SFT policy on human demonstrations, train a reward model from human-ranked model outputs, then fine-tune the policy with PPO against the reward model while using pretraining-mix updates to reduce regressions.

  • Collect prompts from the API distribution plus labeler-written prompts.
  • Have labelers write demonstrations of desired behavior and train SFT models.
  • Sample multiple outputs per prompt and have labelers rank the outputs.
  • Train a 6B reward model to predict the preferred output from ranking data.
  • Optimize the SFT policy with PPO using the reward model as the reward function.
  • Add KL regularization and PPO-ptx pretraining mix to reduce reward over-optimization and downstream regressions.
  • Evaluate with labeler preference ratings, API-distribution metadata, TruthfulQA, RealToxicityPrompts, bias datasets, and public NLP tasks.

Claims

C1 · paper-supported

InstructGPT uses a three-stage RLHF pipeline: SFT on demonstrations, reward modeling from rankings, and PPO against the learned reward model.

Scope: Section 3.1 and Figure 2.

  • E-method: The paper applies three steps: collect demonstrations and train SFT; collect comparison rankings and train a reward model; optimize the SFT policy with PPO using the reward model as a scalar reward.

C2 · paper-supported

The training data is deployment-shaped: about 13k SFT prompts, 33k RM prompts, and 31k PPO prompts, mostly from a commercial API prompt distribution.

Scope: Section 3.2.

  • E-data: The paper reports about 13k SFT training prompts, 33k RM training prompts, and 31k PPO training prompts, drawn primarily from API prompts plus labeler-written prompts; the prompt dataset is over 96% English.

C3 · paper-supported

The paper relies on a small, trained contractor pool for demonstrations, rankings, and evaluation, with measured agreement between training and held-out labelers.

Scope: Section 3.3.

  • E-labelers: About 40 contractors produced demonstrations, comparisons, and evaluations. Training labelers agreed with each other 72.6 +/- 1.5% of the time, and held-out labelers agreed 77.3 +/- 1.3%.

C4 · paper-supported

Human evaluators prefer InstructGPT outputs over GPT-3 outputs on the API prompt distribution, including a 1.3B InstructGPT model over 175B GPT-3.

Scope: Abstract, Figure 1, and main findings.

  • E-preference: The paper reports that 1.3B InstructGPT outputs are preferred to 175B GPT-3 outputs, and that 175B InstructGPT is preferred to 175B GPT-3 85 +/- 3% of the time and to few-shot prompted 175B GPT-3 71 +/- 4% of the time.

C5 · paper-supported

InstructGPT improves truthfulness and reduces some toxicity relative to GPT-3, while not significantly improving bias benchmarks.

Scope: Main findings and Section 3.5.

  • E-truth-toxicity: The paper reports better TruthfulQA behavior, hallucination rates of 21% for InstructGPT versus 41% for GPT-3 on closed-domain API tasks, about 25% fewer toxic outputs when prompted to be respectful, and no significant improvement on Winogender or CrowSPairs.

C6 · paper-supported

Mixing pretraining updates into PPO reduces public NLP regressions without sacrificing the labeler-preference gains.

Scope: Main findings and Section 3.4.

  • E-ppo-ptx: The authors observe public NLP regressions under RLHF and report that PPO-ptx, which mixes pretraining gradients into PPO, greatly reduces those regressions without compromising labeler preference scores.

C7 · paper-supported

The paper is clear that InstructGPT is not fully aligned or fully safe and can become easier to misuse because it follows instructions better.

Scope: Section 5.2, Section 5.3, and OpenAI project page limitations.

  • E-limitations: The limitations state that labeler preferences are not representative of all affected people, data is mostly English, the models still make up facts and generate toxic or biased outputs, and stronger instruction-following can make misuse easier.

Agent Infrastructure Takeaways

  • Post-training is infrastructure. Instruction following depends on data pipelines, labeling operations, reward-model training, and eval loops, not just a better prompt.
  • Train on the distribution where the agent will run. InstructGPT's advantage comes from API-shaped prompts and human preferences rather than only public NLP task mixtures.
  • Preference models are policy surfaces. They encode who labeled, who wrote guidelines, which prompts were collected, and which deployment policies shaped the data.
  • Alignment tax is a runtime economics issue: if safety fine-tuning breaks useful capabilities, builders will route around it unless the tax is measured and mitigated.
  • Better instruction following increases both usefulness and misuse potential, so agent runtimes still need tool policy, refusal behavior, sandboxing, and monitoring.
  • For agents, RLHF is the ancestor of today's trajectory preference and verifiable-reward systems: define the behavior target, collect supervision, optimize, and evaluate on held-out tasks.

Caveats

  • The paper does not make models fully aligned or fully safe; it explicitly says simple mistakes, hallucinations, bias, and unsafe content remain.
  • Human feedback reflects the selected contractor pool, researcher guidance, customer prompts, and API policies, not all user communities.
  • The data is mostly English, so cultural and language generalization are bounded.
  • The strongest reported results are preference judgments on the authors' API prompt distribution, which is exactly the point but also limits external generality.
  • Reward models can be over-optimized; the paper uses KL penalties and PPO-ptx to control this, not to solve it permanently.
  • Instruction following by itself is not an agent safety boundary because it can make harmful instructions easier to carry out.