agent-readings

Multi-Agent Systems

MetaGPT

MetaGPT makes multi-agent software development work more like an assembly line by encoding SOPs, role handoffs, and structured artifacts.

8Claims
8Evidence

Why It Matters

MetaGPT is the strongest SOP argument in the reading list. It treats agent orchestration as workflow engineering, where the payloads are PRDs, designs, APIs, tasks, code, tests, and feedback rather than free-form chat.

Problem

  • Role-playing and chat-based multi-agent systems can drift into idle chatter or inconsistent handoffs.
  • Naively chaining LLMs can amplify hallucinations across agents.
  • Software development requires structured intermediate artifacts, not only conversational agreement.
  • Many agent frameworks lacked precompilation execution, PRD generation, technical design, API interface generation, and full role-based task management.

Key gap: The paper asks whether encoding human software-development SOPs into an LLM multi-agent framework can make collaborative code generation more coherent and executable.

Method

Model software development as a standardized multi-agent assembly line: each role creates structured outputs, publishes them to a shared pool, downstream roles subscribe to prerequisites, and the Engineer iterates using executable tests and feedback.

  • Define agent profiles with name, role, goal, constraints, skills, and context.
  • Run the SOP sequence from requirement analysis through design, task decomposition, code generation, and QA.
  • Require structured artifacts at each handoff instead of unconstrained natural-language dialogue.
  • Store artifacts in a shared message pool and use subscriptions so agents receive relevant dependencies.
  • After initial code generation, execute unit tests or precompilation checks.
  • If errors occur, feed execution results, historical messages, PRD, design, and code back to the Engineer for up to three retries.
  • Evaluate on HumanEval, MBPP, and SoftwareDev.

Claims

C1 · paper-supported

MetaGPT encodes software-development SOPs into prompt sequences and agent workflows to reduce the cascading hallucinations of naive multi-agent chaining.

Scope: Abstract, Introduction, and Section 3.

  • E-sop-framing: The paper frames cascading hallucination as a problem in naive LLM chaining and proposes SOPs, role specialization, structured outputs, and workflow management as a meta-programming framework.

C2 · paper-supported

The core simulated company uses specialized Product Manager, Architect, Project Manager, Engineer, and QA Engineer roles.

Scope: Section 3.1.

  • E-roles: MetaGPT defines Product Manager, Architect, Project Manager, Engineer, and QA Engineer roles, each with profile, goal, constraints, context, and skills.

C3 · paper-supported

MetaGPT replaces much unconstrained dialogue with structured documents and diagrams so downstream agents receive standardized handoff artifacts.

Scope: Section 3.2.

  • E-structured-communication: The paper argues pure natural language can distort information, so agents communicate through role-specific documents and diagrams such as PRDs, system interface designs, sequence flow diagrams, data structures, and API definitions.

C4 · paper-supported

The shared message pool plus role-specific subscriptions is the communication substrate for publishing artifacts and triggering downstream actions.

Scope: Section 3.2.

  • E-message-pool: Agents publish structured messages into a shared message pool and subscribe to role-relevant information; actions activate after prerequisite dependencies are received.

C5 · paper-supported

Executable feedback improves code quality by having the Engineer execute code and tests, use errors plus historical context, and retry debugging up to a bounded limit.

Scope: Section 3.3 and Section 4.4.

  • E-executable-feedback: After code generation, the Engineer executes code and test cases, receives test results, debugs with past messages and artifacts, and retries until tests pass or a maximum of three retries is reached. Executable feedback adds 4.2% on HumanEval and 5.4% on MBPP and reduces SoftwareDev human revision cost from 2.25 to 0.83.

C6 · paper-supported

MetaGPT reports 85.9% and 87.7% Pass@1 on HumanEval and MBPP with GPT-4 collaboration.

Scope: Introduction and Section 4.2.

  • E-code-benchmarks: The paper reports MetaGPT with GPT-4 collaboration achieving 85.9% Pass@1 on HumanEval and 87.7% Pass@1 on MBPP.

C7 · paper-supported

On SoftwareDev, MetaGPT improves executability and reduces human revision cost compared with ChatDev.

Scope: Section 4.2, Table 1.

  • E-softwaredev: On SoftwareDev, ChatDev scores 2.25 executability with 2.5 human revisions, while MetaGPT scores 3.75 executability with 0.83 revisions. ChatDev uses 248.9 tokens per code line; MetaGPT uses 124.3.

C8 · paper-supported

Role ablations show that adding Product Manager, Architect, and Project Manager roles improves executability and reduces revision burden, while increasing cost.

Scope: Section 4.4, Table 3.

  • E-role-ablation: Engineer-only has executability 1.0 and 10 revisions. Adding Product Manager gives 2.0 and 6.5 revisions; adding Architect gives 2.5 and 4.0 revisions; adding all Product, Architect, and Project Manager gives executability 4.0 and 2.5 revisions, with expense rising from $0.915 to $1.385.

Agent Infrastructure Takeaways

  • Multi-agent collaboration needs artifacts, not just messages. PRDs, designs, APIs, tasks, tests, and code are the state transferred between agents.
  • SOPs are control planes. They define who acts, what they emit, who consumes it, and when downstream work can start.
  • Publish-subscribe beats point-to-point chat for complex workflows because dependencies are explicit and reusable.
  • Executable feedback is the load-bearing verifier. Code review by another LLM is weaker than running tests in a sandbox.
  • More agents help only when they correspond to real workflow responsibilities and produce useful structured outputs.
  • The runtime must record artifacts and provenance so later agents can debug against the PRD, design, task, and previous execution results.

Caveats

  • The current arXiv version is v7 from 2024, while the reading-list entry anchors the original 2023 paper.
  • MetaGPT focuses on software-development workflows; claims should not be generalized to arbitrary multi-agent systems without evidence.
  • The SoftwareDev benchmark is self-generated and only seven representative tasks are used for some comparisons.
  • The framework requires safe code execution infrastructure, but the paper does not deeply analyze sandboxing.
  • Structured outputs can reduce drift but cannot make hallucinated requirements or designs correct by themselves.
  • The open-source project has evolved and moved from the original geekan/MetaGPT URL to FoundationAgents/MetaGPT.