Why It Matters
WebGPT predates most agent frameworks but already has the shape of a serious agent: a constrained tool interface, trajectory demonstrations, preference optimization, citations for audit, and a threat model for side effects.
Problem
- Long-form QA systems needed both retrieval and synthesis, and prior methods often optimized document retrieval separately from answer quality.
- Language models hallucinated obscure factual claims because pretraining alone did not force grounding in current sources.
- Human evaluation of factual accuracy is hard when model answers do not expose the evidence behind their claims.
- Browser-use models needed a task setup humans could perform so demonstrations and comparisons could be collected.
Key gap: The paper asks how to train a language model to use a web browser for open-ended QA and make factuality evaluation tractable through collected references.
Method
Expose a restricted text browser to GPT-3, collect human browsing demonstrations and answer comparisons on ELI5 questions, train with behavior cloning and reward modeling, then use best-of-n rejection sampling or PPO to optimize answer quality.
- Represent browser state as text: question, page title, visible page text, cursor state, prior actions, and action budget.
- Collect human demonstrations of using the browser to answer long-form ELI5 questions.
- Collect human comparisons between two model-generated cited answers.
- Fine-tune GPT-3 with behavior cloning on browser-command demonstrations.
- Train reward models from answer comparisons.
- Optimize with PPO or, for the best reported model, sample many behavior-cloned answers and select the reward-model winner.
- Evaluate against human demonstrators, ELI5 reference answers, and TruthfulQA.
Claims
C1 · paper-supported
WebGPT turns web research into a constrained text-browser action loop that both humans and models can operate.
Scope: Section 2 and Table 1.
- E-browser-actions: The model receives a text summary of browser state and must issue one valid command: search, click, find, quote, scroll, top, back, or end. Invalid actions count against the action budget.
C2 · paper-supported
The paper collects human demonstrations and model-output comparisons, mostly on ELI5, to train browser use and answer preference.
Scope: Section 3.1.
- E-data: The authors collected around 6,000 browser demonstrations, 92% for ELI5 questions, and around 21,500 comparisons, 98% for ELI5 questions.
C3 · paper-supported
The best reported WebGPT model uses behavior cloning followed by reward-model best-of-n rejection sampling rather than PPO alone.
Scope: Abstract, Section 3.2, and Section 5.1.
- E-training: Training methods include behavior cloning on demonstrations, reward modeling from comparisons, PPO against the reward model with a KL penalty, and best-of-n rejection sampling against the reward model.
- E-rejection-sampling: The main evaluated models are behavior-cloned policies with best-of-n rejection sampling: 760M best-of-4, 13B best-of-16, and 175B best-of-64. The paper reports 175B best-of-64 BC preferred 68% of the time to 175B BC, while 175B RL is preferred 58% of the time to 175B BC.
C4 · paper-supported
On ELI5, the 175B best-of-64 WebGPT model is preferred to human browser demonstrators 56% of the time and to highest-voted Reddit answers 69% of the time.
Scope: Abstract and Section 4.1.
- E-eli5: The best 175B best-of-64 model is preferred 56% of the time to human demonstrator answers and 69% of the time to highest-voted ELI5 Reddit answers.
C5 · paper-supported
WebGPT improves TruthfulQA over GPT-3 but remains below human performance on out-of-distribution adversarial questions.
Scope: Abstract, Section 4.2, and project page.
- E-truthfulqa: The best model's TruthfulQA answers are true 75% of the time and both true and informative 54% of the time; all WebGPT models outperform GPT-3 prompts but remain below humans.
C6 · paper-supported
Collected references make factuality evaluation more inspectable, but citations can also create overreliance and reward cherry-picked evidence.
Scope: Section 6.2 and Section 6.4.
- E-reference-benefits: The paper argues references make feedback more accurate and less noisy because labelers can judge whether claims are supported by cited sources, and they make browsing trajectories inspectable.
- E-reference-risks: The paper warns that citations can make answers appear more authoritative, cause overreliance, and incentivize models to cherry-pick references that persuade labelers rather than fairly represent evidence.
C7 · paper-supported
Even restricted web access creates train-time and inference-time side-effect risks that grow with model capability.
Scope: Section 6.5.
- E-web-risk: The authors discuss live web access risks at train and inference time, including hypothetical side effects such as editing a source to create convincing evidence, and say safety burden should rise with model capability.
Agent Infrastructure Takeaways
- WebGPT is an agent paper before the term stabilized: observation, action grammar, environment transition, evidence collection, and final answer.
- Tool interfaces should be constrained and logged. The text browser makes invalid actions explicit and preserves a trajectory humans can inspect.
- Citations are not decoration; they are training infrastructure that lets humans evaluate factual support without doing unrestricted research from scratch.
- Reward-model optimization can move from training time to inference time. Best-of-n is expensive, but effective when the environment is unpredictable.
- A browser is a side-effect boundary. Even read-mostly web access creates risks once the model can search, click, and choose evidence strategically.
- For agent evaluation, judge the trajectory as well as the final answer: sources, search path, quote selection, and synthesis errors are separate failure modes.
Caveats
- The system is trained primarily on ELI5 long-form QA, so it is not a general-purpose web automation benchmark.
- The best model uses substantial inference-time compute through best-of-64 sampling.
- The browser is restricted; conclusions do not directly transfer to agents with forms, authenticated sites, or write actions.
- Citations can increase perceived truthfulness even when the answer is wrong or the source is unreliable.
- The model can reinforce biases from GPT-3, the web, source selection, and question framing.
- The paper reports out-of-distribution brittleness on TruthfulQA and examples of unreliable-source failures.