agent-readings

<div class="slide-frame slide-frame--center">
  <p class="deck-kicker">agent-readings / Evaluation and Benchmarks</p>
  <h1>GAIA</h1>
  <p class="deck-subtitle">Simple answers, hard tool paths</p>

<div class="deck-meta-strip">
    <div class="deck-meta-pill"><strong>2023</strong><span>arXiv 2023</span></div>
    <div class="deck-meta-pill"><strong>6</strong><span>claims</span></div>
    <div class="deck-meta-pill"><strong>6</strong><span>evidence refs</span></div>
    <div class="deck-meta-pill"><strong>paper-read</strong><span>status</span></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">The one sentence</p>
  <h2>GAIA evaluates general AI assistants with concise, factoid real-world questions that are easy for humans but require AI systems to browse, reason, handle multimodal evidence, and use tools accurately.</h2>
  <p class="deck-note">For agents, GAIA is a clean benchmark of compound assistant behavior: the answer is usually short, but reaching it requires correct tool use, web navigation, evidence synthesis, and exact formatting.</p>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Before this paper</p>
  <h2>The friction was structural.</h2>
  <div class="apple-card-grid apple-card-grid--three">
    <div class="apple-card"><p>Many LLM benchmarks were becoming saturated or were vulnerable to data contamination.</p></div>
<div class="apple-card"><p>Some benchmarks target tasks that are difficult for humans, even though current AI systems can already outperform non-experts on professional exams.</p></div>
<div class="apple-card"><p>Open-ended generation evaluation can require expensive human or model-based judging.</p></div>
  </div>
  <div class="deck-callout">
    <span>Key gap</span>
    <p>The paper asks how to evaluate general assistant robustness on practical tasks whose solutions are easy to verify but hard for AI systems to execute reliably.</p>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Core idea</p>
  <h2>Craft questions that require real-world tool use and multi-step evidence gathering, make final answers short and unique, collect human baselines, and publish a...</h2>
  <p class="deck-note">Craft questions that require real-world tool use and multi-step evidence gathering, make final answers short and unique, collect human baselines, and publish a split with held-out answers for leaderboard evaluation.</p>
  <div class="idea-loop" aria-label="Agent loop">
    <div><strong>Model</strong><span>reason</span></div>
    <div><strong>Runtime</strong><span>act</span></div>
    <div><strong>World</strong><span>observe</span></div>
  </div>
  <div class="step-strip">
    <div class="step-item">
  <span>01</span>
  <p>Design questions that are simple for humans but require tool-use, browsing, reasoning, or multimodal handling for AI.</p>
</div>
<div class="step-item">
  <span>02</span>
  <p>Ensure answers are concise, factual, unique, and unlikely to appear directly in training data.</p>
</div>
<div class="step-item">
  <span>03</span>
  <p>Group tasks into levels of increasing difficulty.</p>
</div>
<div class="step-item">
  <span>04</span>
  <p>Collect human respondent performance.</p>
</div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Action grammar</p>
  <h2>The agent is an interface contract.</h2>
  <div class="apple-card-grid apple-card-grid--two">
    <div class="apple-card"><p>question: real-world prompt with one concise factual answer.</p></div>
<div class="apple-card"><p>evidence: optional image, document, web page, or clue used to answer.</p></div>
<div class="apple-card"><p>tools: browser, search, code, calculator, multimodal parser, or other assistant capability.</p></div>
<div class="apple-card"><p>answer string: final concise response checked against ground truth.</p></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Executable shape</p>
  <h2>The mechanism should fit on one screen.</h2>

```python
for question in gaia:
    evidence = question.attachments_or_web_context
    agent_trace = agent.solve(question.prompt, tools=[browser, code, calculator, vision])
    answer = normalize(agent_trace.final_answer)
    success = exact_or_rule_match(answer, question.ground_truth)
    record(success, level=question.level)
```
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Evidence</p>
  <h2>The proof objects.</h2>
  <div class="proof-grid">
    <div class="proof-card">
  <span>E-question-count</span>
  <p>The paper states GAIA comprises 466 carefully crafted questions and answers.</p>
</div>
<div class="proof-card">
  <span>E-required-abilities</span>
  <p>The paper says GAIA tasks require reasoning, multimodality, web browsing, and tool-use proficiency, with examples involving NIH pages, Wikipedia calculations, and NASA/APOD image...</p>
</div>
<div class="proof-card">
  <span>E-factoid-answers</span>
  <p>GAIA questions are designed to have unique, factual, concise, unambiguous answers so evaluation is simple and robust.</p>
</div>
<div class="proof-card">
  <span>E-human-gpt4</span>
  <p>The paper reports 92% human performance versus 15% for GPT-4 equipped with plugins.</p>
</div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Claim map</p>
  <h2>What the review actually supports.</h2>
  <div class="claim-grid">
    <div class="claim-card">
  <span>C1 · paper-supported</span>
  <p>GAIA contains 466 carefully crafted questions targeting general AI assistant abilities.</p>
  <small>E-question-count</small>
</div>
<div class="claim-card">
  <span>C2 · paper-supported</span>
  <p>GAIA questions require abilities such as reasoning, multimodal handling, web browsing, and tool-use proficiency.</p>
  <small>E-required-abilities</small>
</div>
<div class="claim-card">
  <span>C3 · paper-supported</span>
  <p>GAIA uses concise, unambiguous factual answers to support simple automatic evaluation.</p>
  <small>E-factoid-answers</small>
</div>
<div class="claim-card">
  <span>C4 · paper-supported</span>
  <p>Human respondents score 92% on GAIA, while GPT-4 equipped with plugins scores 15% in the paper's headline comparison.</p>
  <small>E-human-gpt4</small>
</div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Agent infrastructure</p>
  <h2>What changes if you build systems.</h2>
  <div class="apple-card-grid apple-card-grid--two">
    <div class="apple-card"><p>GAIA is a clean assistant benchmark because the final answer is short but the path is operationally complex.</p></div>
<div class="apple-card"><p>Tool-use evaluation should ask whether the tool was needed and used correctly, not merely whether the model can name a tool.</p></div>
<div class="apple-card"><p>Concise factoid answers make grading cheaper and less judge-dependent than free-form reports.</p></div>
<div class="apple-card"><p>The human-vs-AI gap is useful because these are not expert-only tasks; they measure robustness on ordinary information work.</p></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Caveats</p>
  <h2>What this does not prove.</h2>
  <div class="apple-card-grid apple-card-grid--two">
    <div class="apple-card"><p>The benchmark is small and curated.</p></div>
<div class="apple-card"><p>Paper-era GPT-4/plugin results are historical, not current SOTA.</p></div>
<div class="apple-card"><p>Exact answer matching can be brittle if normalization rules are underspecified.</p></div>
<div class="apple-card"><p>Some tasks depend on web state that can drift unless evidence is fixed or reproduced.</p></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Run it</p>
  <h2>No PoC yet.</h2>
  <p class="deck-note">No PoC yet - contributions welcome. A tiny PoC could define three GAIA-style questions with fixed evidence and exact answer matching.</p>
  <div class="reference-list">
    <ul><li>Paper: <a href="https://arxiv.org/abs/2311.12983">GAIA: A Benchmark for General AI Assistants</a></li><li>Project/code: <a href="https://huggingface.co/gaia-benchmark">https://huggingface.co/gaia-benchmark</a></li><li>complements: <a href="https://arxiv.org/abs/2307.13854">WebArena</a></li><li>tool_use: <a href="https://arxiv.org/abs/2302.04761">Toolformer</a></li><li>broader_harness: <a href="https://arxiv.org/abs/2308.03688">AgentBench</a></li></ul>
  </div>
</div>

Note: This deck is synthesized from `data/reviews/gaia-mialon-2023.json`. Update the review record, then run `bun run build`.