agent-readings · HuggingGPT

<div class="slide-frame slide-frame--center">
  <p class="deck-kicker">agent-readings / Tool Use and Interoperability</p>
  <h1>HuggingGPT</h1>
  <p class="deck-subtitle">ChatGPT as controller for specialist model execution</p>

<div class="deck-meta-strip">
    <div class="deck-meta-pill"><strong>2023</strong><span>NeurIPS 2023</span></div>
    <div class="deck-meta-pill"><strong>6</strong><span>claims</span></div>
    <div class="deck-meta-pill"><strong>6</strong><span>evidence refs</span></div>
    <div class="deck-meta-pill"><strong>paper-read</strong><span>status</span></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">The one sentence</p>
  <h2>HuggingGPT uses ChatGPT as a controller that plans tasks, selects Hugging Face expert models, executes subtasks, and summarizes the results.</h2>
  <p class="deck-note">HuggingGPT is an early clean statement of the controller-plus-experts pattern: the hard infrastructure is not one model call, but task graphs, resource dependencies, model routing, execution endpoints, and result integration.</p>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Before this paper</p>
  <h2>The friction was structural.</h2>
  <div class="apple-card-grid apple-card-grid--three">
    <div class="apple-card"><p>LLMs such as ChatGPT were strong at text reasoning and interaction but could not directly process many vision, audio, video, or specialized ML tasks.</p></div>
<div class="apple-card"><p>ML communities such as Hugging Face hosted many expert models, but those models did not autonomously coordinate to solve complex user requests.</p></div>
<div class="apple-card"><p>Complex multimodal requests often decompose into ordered subtasks with dependencies across generated resources.</p></div>
  </div>
  <div class="deck-callout">
    <span>Key gap</span>
    <p>The paper asks whether an LLM can serve as the controller that plans, routes, executes, and summarizes across many expert models using language descriptions as the shared interface.</p>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Core idea</p>
  <h2>Parse the user request into a structured task graph, select an expert model for each task from Hugging Face descriptions, execute independent tasks in parallel...</h2>
  <p class="deck-note">Parse the user request into a structured task graph, select an expert model for each task from Hugging Face descriptions, execute independent tasks in parallel and dependent tasks in order, then feed structured execution results back to ChatGPT for response...</p>
  <div class="idea-loop" aria-label="Agent loop">
    <div><strong>Model</strong><span>reason</span></div>
    <div><strong>Runtime</strong><span>act</span></div>
    <div><strong>World</strong><span>observe</span></div>
  </div>
  <div class="step-strip">
    <div class="step-item">
  <span>01</span>
  <p>Use ChatGPT to parse a user request into task names, ids, dependencies, and modality-specific arguments.</p>
</div>
<div class="step-item">
  <span>02</span>
  <p>For each planned task, filter candidate expert models by task type.</p>
</div>
<div class="step-item">
  <span>03</span>
  <p>Rank candidate models by Hugging Face downloads and ask ChatGPT to select the most appropriate model from descriptions.</p>
</div>
<div class="step-item">
  <span>04</span>
  <p>Execute tasks through selected models, replacing resource placeholders with outputs from prerequisite tasks.</p>
</div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Action grammar</p>
  <h2>The agent is an interface contract.</h2>
  <div class="apple-card-grid apple-card-grid--two">
    <div class="apple-card"><p>Task plan: a JSON-like list of tasks with task name, id, dependency ids, and arguments.</p></div>
<div class="apple-card"><p>Dependency handle: a resource placeholder such as <resource>-task_id that points to an output generated by an earlier...</p></div>
<div class="apple-card"><p>Candidate model: a Hugging Face model description filtered by task type and ranked for selection.</p></div>
<div class="apple-card"><p>Model assignment: a JSON-like model id plus rationale for the selected expert model.</p></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Executable shape</p>
  <h2>The mechanism should fit on one screen.</h2>

```python
tasks = chatgpt_plan(user_request, chat_logs)
for task in topological_order(tasks):
    candidates = top_models_by_task(task.name)
    model = chatgpt_select_model(task, candidates)
    args = resolve_resources(task.args, prior_outputs)
    outputs[task.id] = execute_model(model, args)
return chatgpt_summarize(user_request, tasks, outputs)
```
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Evidence</p>
  <h2>The proof objects.</h2>
  <div class="proof-grid">
    <div class="proof-card">
  <span>E-four-stage</span>
  <p>The paper defines HuggingGPT as an LLM-powered agent where ChatGPT conducts task planning, selects Hugging Face models by descriptions, executes subtasks with selected models,...</p>
</div>
<div class="proof-card">
  <span>E-dependencies</span>
  <p>Task plans include task name, id, dependency ids, and arguments. HuggingGPT uses resource placeholders such as <resource>-task_id to pass generated resources between tasks and...</p>
</div>
<div class="proof-card">
  <span>E-model-selection</span>
  <p>HuggingGPT gathers model descriptions from the model hub, filters candidates by task type, ranks them by Hugging Face downloads to fit context limits, and asks the LLM to select...</p>
</div>
<div class="proof-card">
  <span>E-planning-eval</span>
  <p>GPT-3.5 gets 52.62 single-task accuracy and 54.45 F1, 0.54 sequential edit distance and 51.92 F1, and 50.48 GPT-4 score and 51.91 F1 for graph tasks. Alpaca-7B and Vicuna-7B are...</p>
</div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Claim map</p>
  <h2>What the review actually supports.</h2>
  <div class="claim-grid">
    <div class="claim-card">
  <span>C1 · paper-supported</span>
  <p>HuggingGPT organizes model collaboration into four stages: task planning, model selection, task execution, and response generation.</p>
  <small>E-four-stage</small>
</div>
<div class="claim-card">
  <span>C2 · paper-supported</span>
  <p>Task planning represents dependencies explicitly with task ids and resource handles, allowing generated artifacts to flow into later subtasks.</p>
  <small>E-dependencies</small>
</div>
<div class="claim-card">
  <span>C3 · paper-supported</span>
  <p>Model selection is driven by natural-language model descriptions, filtered by task type and narrowed by popularity because all candidate models cannot fit in...</p>
  <small>E-model-selection</small>
</div>
<div class="claim-card">
  <span>C4 · paper-supported</span>
  <p>In task-planning evaluations, GPT-3.5 substantially outperforms Alpaca-7B and Vicuna-7B across single, sequential, and graph task planning.</p>
  <small>E-planning-eval</small>
</div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Agent infrastructure</p>
  <h2>What changes if you build systems.</h2>
  <div class="apple-card-grid apple-card-grid--two">
    <div class="apple-card"><p>A model-routing agent needs a typed task graph, not just a chat transcript.</p></div>
<div class="apple-card"><p>Resource handles are the minimal runtime abstraction for passing generated artifacts between model calls.</p></div>
<div class="apple-card"><p>Model descriptions are API documentation for model selection; they need curation, ranking, compression, and freshness checks.</p></div>
<div class="apple-card"><p>The controller/expert split makes endpoint health, latency, local fallback, and parallel execution core infrastructure concerns.</p></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Caveats</p>
  <h2>What this does not prove.</h2>
  <div class="apple-card-grid apple-card-grid--two">
    <div class="apple-card"><p>The paper's strongest claims are mostly architectural and qualitative; it does not provide a large executable benchmark of end-to-end task success.</p></div>
<div class="apple-card"><p>Task-planning evaluation ignores task arguments in the main quantitative setup.</p></div>
<div class="apple-card"><p>Human-annotated planning evaluation has only 46 complex examples.</p></div>
<div class="apple-card"><p>Model selection by downloads is pragmatic but can select popular rather than correct or safe models.</p></div>
  </div>
</div>

---

<div class="slide-frame">
  <p class="deck-kicker">Run it</p>
  <h2>No PoC yet.</h2>
  <p class="deck-note">No PoC yet - contributions welcome. A useful local PoC would route a small set of image/text/audio tasks across mocked local expert endpoints using a typed task graph and resource handles.</p>
  <div class="reference-list">
    <ul><li>Paper: <a href="https://arxiv.org/abs/2303.17580">HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face</a></li><li>Project/code: <a href="https://github.com/microsoft/JARVIS">https://github.com/microsoft/JARVIS</a></li><li>contrasts: <a href="https://arxiv.org/abs/2302.04761">Toolformer</a></li><li>connects-to: <a href="https://arxiv.org/abs/2210.03629">ReAct</a></li><li>precedes: <a href="https://arxiv.org/abs/2308.08155">AutoGen</a></li></ul>
  </div>
</div>

Note: This deck is synthesized from `data/reviews/hugginggpt-shen-2023.json`. Update the review record, then run `bun run build`.