---
title: 【DL輪読会】“Multi-Agent LLM Systems: Do They Work, Which Structures Help, and Do They Beat a Single Agent?”
tags: 
author: [Deep Learning JP](https://www.docswell.com/user/DeepLearning2023)
site: [Docswell](https://www.docswell.com/)
thumbnail: https://bcdn.docswell.com/page/5EG1G5G4JL.jpg?width=480
description: 【DL輪読会】“Multi-Agent LLM Systems: Do They Work, Which Structures Help, and Do They Beat a Single Agent?” by Deep Learning JP
published: November 05, 26
canonical: https://www.docswell.com/s/DeepLearning2023/KY8J33-2026-11-09-110610
---
# Page. 1

![Page Image](https://bcdn.docswell.com/page/5EG1G5G4JL.jpg)

DEEP LEARNING JP
[DL Papers]
“Multi-Agent LLM Systems: Do They Work, Which
Structures Help, and Do They Beat a Single Agent?”
Presenter: Tsogbadrakh Khangai, Matsuo-Iwasawa Lab M1
http://deeplearning.jp/
1


# Page. 2

![Page Image](https://bcdn.docswell.com/page/4JQDXZXR7P.jpg)

Background: Multi-agent LLM systems
Single agent (SAS)
Multi-agent (MAS)
Planner
LLM
Coder
think, act, observe,
repeat
Reviewer
messages, feedback loops
tools / env
tools / env
Three mechanisms a team is supposed to add
Decomposition
split a big task into parts
Specialization
each role does what it is good at
Verification
another agent checks the output
2


# Page. 3

![Page Image](https://bcdn.docswell.com/page/K74Z636QE1.jpg)

Three MAS papers
Paper
1
Question
What they study
Do they work, and when they fail, why
Hand-built open-source frameworks (MetaGPT,
ChatDev, AG2 ...)
Kim et al., Google Research / DeepMind / MIT. Published as
“Capable language models can outgrow the benefits of
collaboration”, Nature Machine Intelligence 2026
Which structures are actually good
Five canonical topologies built by the authors,
matched token budgets
The Illusion of Multi-Agent Advantage
Even with a good structure, does it beat the
single agent you already have
Automatically generated MAS (ADAS, AFlow,
DyLAN ...) vs. self-consistency
Why Do Multi-Agent LLM Systems Fail?
Cemri et al., UC Berkeley, NeurIPS 2025
Towards a Science of Scaling Agent Systems
2
3
Jwalapuram et al., Salesforce Research, NeurIPS 2026
3


# Page. 4

![Page Image](https://bcdn.docswell.com/page/LJ1RV1VLJG.jpg)

MAS failures, and what they are tied to
PAPER 1: WHY DO MULTI-AGENT LLM
SYSTEMS FAIL?
4


# Page. 5

![Page Image](https://bcdn.docswell.com/page/GJW1L8LQE2.jpg)

Setup: 1,642 traces, 7 frameworks
Data
MAST-Data: 1,642 annotated execution traces, 7 frameworks (MetaGPT, ChatDev, HyperAgent, AppWorld, AG2, Magentic-One,
OpenManus)
Models
GPT-4 / GPT-4o, Claude 3.7 Sonnet, Qwen2.5-Coder-32B, CodeLlama-7B
Taxonomy
Grounded Theory on 150 traces, 6 expert annotators, human agreement κ = 0.88
Scaling
LLM-as-judge with o1 (few-shot): 94% accuracy, κ = 0.77 vs. humans, ~$1.8 per trace
Link
arxiv.org/abs/2503.13657
Benchmarks in plain words
ProgramDev
write a small program from a spec
AppWorld
operate phone apps through APIs
HumanEval
short Python functions
OlympiadBench
olympiad math problems
SWE-Bench Lite
fix real GitHub issues
GAIA
assistant questions that need web and tools
5


# Page. 6

![Page Image](https://bcdn.docswell.com/page/4EZPM8M3J3.jpg)

Results: MAS failed in 41% to 86.7% of runs
0
AG2 (OlympiadBench)
MetaGPT (ProgramDev)
10
20
30
40
50
60
70
80
90
41.0%
100
Analysis
• Each system is on a different benchmark, so
this is not a ranking
• Models: GPT-4o and Claude 3.7 Sonnet
60.0%
• Failure = task not completed as specified
• Harder benchmarks had higher failure rates
Magentic-One (GAIA)
ChatDev (ProgramDev)
HyperAgent (SWE-Bench Lite)
AppWorld (Test-C)
62.0%
66.7%
74.7%
86.7%
Figure 5 of the paper
6


# Page. 7

![Page Image](https://bcdn.docswell.com/page/Y76MNPNZEV.jpg)

Results: MAST 14 failure modes in 3 categories
Category
FC1 System design issues
44.2% of failures
FC2 Inter-agent misalignment
32.3% of failures
FC3 Task verification
23.5% of failures
Failure mode
Share
1.1 Disobey task specification
11.8%
1.2 Disobey role specification
1.5%
1.3 Step repetition
15.7%
1.4 Loss of conversation history
2.8%
1.5 Unaware of termination conditions
12.4%
2.1 Conversation reset
2.2%
2.2 Fail to ask for clarification
6.8%
2.3 Task derailment
7.4%
2.4 Information withholding
0.8%
2.5 Ignored other agent’s input
1.9%
2.6 Reasoning-action mismatch
13.2%
3.1 Premature termination
6.2%
3.2 No or incomplete verification
8.2%
3.3 Incorrect verification
9.1%
The three largest
• Step repetition 15.7%
• Reasoning-action mismatch 13.2%
• Unaware of termination 12.4%
Shares are percentages of all observed failures across 1,642 traces, not failure rates. Figure 1 of the paper
7


# Page. 8

![Page Image](https://bcdn.docswell.com/page/G75Z5K59J4.jpg)

Results: Both model and architecture matter
Model matters
Architecture matters
• Inside MetaGPT, GPT-4o has 39% fewer system-design failures
than Claude 3.7 Sonnet
• Same task: MetaGPT has 60 to 68% fewer FC1 / FC2 failures than
ChatDev, but 1.56× more FC3
• Qwen2.5-Coder-32B is far more robust than CodeLlama-7B
• Failures shift between categories rather than disappear
Interventions guided by MAST (task success, %)
ChatDev ProgramDev-v0
ChatDev HumanEval
AG2 GSM-Plus (GPT-4)
AG2 GSM-Plus (GPT-4o)
Baseline
25.0
89.6
84.75
84.25
Improved prompt
34.4
90.3
89.75
89.00
New topology
40.6
91.5
85.50
88.83
8


# Page. 9

![Page Image](https://bcdn.docswell.com/page/9J2RNWN57R.jpg)

Doubts
• Topology change works in one of two experiments AG2 with GPT-4 shows no significant gain (p = 0.4) and ties with
prompt tuning on GPT-4o
• Taxonomy comes from 150 traces, five frameworks, 2024-era models whether the mode frequencies hold for 2026
models is untested, as the authors note
• Fixes reduce failures, they do not make MAS good ProgramDev completion after the best intervention is still 40.6%
Handoff failures are mostly organizational, and changing the topology helped more than changing the prompt. So which topologies are
worth building?
9


# Page. 10

![Page Image](https://bcdn.docswell.com/page/DEYDPLP6EM.jpg)

Which architectures are actually good?
PAPER 2: TOWARDS A SCIENCE OF
SCALING AGENT SYSTEMS
10


# Page. 11

![Page Image](https://bcdn.docswell.com/page/VJN654M1J8.jpg)

Setup: 5 architectures, 6 tasks
Single agent
Independent
A
A
A
Centralized
Hybrid
A
O
O
A
A
one loop
Decentralized
parallel, no talking
A
everyone debates
A
A
A
orchestrator + workers
A
A
A
orchestrator + side talk
Models
GPT-5 nano / mini / GPT-5, Gemini 2.0 Flash / 2.5 Flash / 2.5 Pro, Claude Sonnet 3.7 / 4 / 4.5
Control
260 configurations, same tools and prompts, matched maximum token budget (team shares one budget, mean ~4,800 reasoning tokens per
trial)
Link
arxiv.org/abs/2512.08296
Benchmarks in plain words
BrowseComp-Plus
find obscure facts by browsing the web
Workbench
office tasks with 16 tools (email, calendar ...)
Finance-Agent
analysis questions over financial filings
SWE-bench Verified
fix real GitHub issues
PlanCraft
Minecraft item crafting from an inventory
Terminal-Bench
command-line tasks
11


# Page. 12

![Page Image](https://bcdn.docswell.com/page/YE9LYQNY73.jpg)

Results: best architecture depends on the task
Benchmark
SAS mean
Best MAS vs. SAS
Worst MAS vs. SAS
Why (authors’ trace reading)
Finance-Agent
0.349
Centralized +80.8%
Independent +57%
parallel information streams
BrowseComp-Plus
0.318
Decentralized +9.2%
Independent −35%
high-entropy search, needs cross-checks
Workbench
0.629
Decentralized +5.6%
−11%
16 tools, coordination overhead
Terminal-Bench
0.344
Independent +1.7%
Centralized −19.2%
2 tools, orchestration not justified
SWE-bench Verified
0.522
Hybrid −2.1%
Independent −14.9%
strong single-agent baseline
PlanCraft
0.568
Hybrid −39.1%
Independent −70.0%
sequential 3-step path, no parallelism
−0.3%
mean change across all MAS
[−58.7%, +77.2%]
Gains when the task splits into parallel streams
Losses when the task is a short sequential path and
coordination only adds turns
95% confidence interval
Relative change = (mean MAS − mean SAS) / mean SAS. Figure 2 and Section 4.1 of the paper
12


# Page. 13

![Page Image](https://bcdn.docswell.com/page/GE8X6GKKJD.jpg)

Results: Capability ceiling exists
~45%
17.2× → 4.4×
Tools < 5
Capability ceiling
Error amplification
Tool-coordination trade-off
Above ~45% single-agent accuracy, adding
agents tends to hurt
The only regularity that survives multiplecomparison correction
Independent agents amplify trace errors
17.2×, centralized verification holds it to
4.4×. Aggregation without a check lets
errors through
More tools, larger coordination penalty.
Tasks with four or fewer tools show little
effect
Predictive model
• Picks the best architecture for 87% of held-out configurations (random: 20%, capability-only: 54%)
• Cross-validated R² = 0.37, so it ranks architectures but does not predict absolute scores across domains
13


# Page. 14

![Page Image](https://bcdn.docswell.com/page/LEL89GZP7R.jpg)

Doubts
• Budget matching cuts both ways fair on cost, but each agent in a 5-agent team gets a fifth of the tokens. The
capability ceiling may partly be a starvation effect on hard tasks
• Six benchmarks means six clusters dataset-level claims are called “directional” by the authors themselves
• R² = 0.37 is a weak model trust the 87% architecture-selection number, not the absolute predictions
Handoff a hand-designed team wins on some tasks. But most people will not hand-design one. They will use a framework that builds the MAS for
them, or just sample one model a few times. How does MAS compare to that?
14


# Page. 15

![Page Image](https://bcdn.docswell.com/page/4JM6ZQV2EW.jpg)

Does MAS beat the single agent you already have?
PAPER 3: THE ILLUSION OF MULTIAGENT ADVANTAGE
15


# Page. 16

![Page Image](https://bcdn.docswell.com/page/PJRP6825E9.jpg)

Setup: automatic MAS vs. CoT-SC
Single-agent baseline
CoT and CoT-SC (ask once, sample 5, majority vote). CoT-SC is the main comparison
Automatic MAS
Inference-time: DyLAN (top-K agent selection), MAS-Zero (meta-agent + verifier)
Pre-optimized: ADAS, AFlow (workflow search), MaAS (supernet sampling), MAS-Orchestra (RL-trained orchestrator)
Backbones
GPT-4o, GPT-5, GPT-OSS-120B, Gemini 2.5 Pro. Cost reported in dollars, not matched
Link
arxiv.org/abs/2606.13003
Benchmarks in plain words
GPQA-Diamond
PhD-level science, multiple choice
BrowseComp-Plus
find obscure facts by browsing
HLE-Maths
Humanity’s Last Exam, math subset
SMFR (new)
synthetic multi-hop financial reasoning over per-investor
transaction records, built to have MAS-friendly structure
SWE-Bench Lite
fix real GitHub issues
16


# Page. 17

![Page Image](https://bcdn.docswell.com/page/PEX3M8DXJX.jpg)

Results: sampling, not coordination
Opening the box
GPT-5
Accuracy
Cost ($)
GPQA-Diamond CoT-SC
87.35%
46
• DyLAN agents vote unanimously in >90% of GPT-5 cases. Expert roles make
no difference (53.4 vs. 54.4 generic)
GPQA-Diamond ADAS
85.23%
832
BrowseComp-Plus CoT-SC
83.92%
67
• MAS-Zero verifier picks the first candidate >45% of the time. Later workers
are “expensive witnesses”
BrowseComp-Plus MAS-Zero
71.63%
1,370
Same backbone, same benchmark: CoT-SC matches or beats every
automatic MAS on the standard benchmarks, at roughly 10× lower cost
• AFlow 7 of 14 final workflows are repeat-one-prompt-and-aggregate
• MaAS 74% of BrowseComp-Plus calls collapse to a single I/O step
• MAS-Orchestra picks Debate agents 85% of the time on GPQA regardless
of difficulty
The committee is five clones of one person
No decomposition, no specialization, just more samples
Exception the authors state: on HLE-Maths, GPT-5 and Gemini 2.5 Pro get significant MAS gains (GPT-5 CoT-SC 33.9% vs. MAS-Zero 38.2%). Table 4 and Section 5 of the paper
17


# Page. 18

![Page Image](https://bcdn.docswell.com/page/3EKYGKX9JD.jpg)

Results: A deliberately designed MAS still wins
Expert-MAS on SMFR, GPT-5
Same design, other backbones
57.0%
96.5%
CoT-SC
Expert-MAS
• Cost comparable: $555 vs. $478 for CoT-SC
• Hand-built: meta-agent parses the problem, extractor and calculator agents
handle sub-tasks, deterministic Python code orchestrates
Backbone
CoT-SC
Expert-MAS
GPT-5
57.0%
96.5%
GPT-OSS-120B
26.1%
36.1%
GPT-4o
22.1%
22.6%
On GPT-4o the structure cannot rescue calculation and retrieval failures.
The authors call this a competency floor
• Strict context separation, no “orchestration noise”
The paper’s own conclusion automatic MAS is an illusion, deliberate MAS needs a strong enough base model and a task with real structure
18


# Page. 19

![Page Image](https://bcdn.docswell.com/page/L739NZ5975.jpg)

Doubts
• The authors built both the benchmark and the system that aces it SMFR and Expert-MAS were designed together.
96.5% is a proof of existence, not a general result
• Dollar cost mixes architecture cost with vendor pricing token counts would separate the two
• Tool parity is not stated Expert-MAS runs a deterministic Python executor and calculator agents. Whether CoT-SC
had the same tools is not in the paper. If not, part of 57 → 96.5 is tools, not coordination
Consistent with paper 2 under a matched budget, paper 2 shows the sign of the MAS effect depends on task structure. Without cost matching,
paper 3 shows the sign looks positive for the wrong reason: sampling volume. Together, a MAS comparison without cost control proves nothing
19


# Page. 20

![Page Image](https://bcdn.docswell.com/page/87DGWRD87G.jpg)

Summary: agreement, and one open question
They agree
They seem to contradict
• The default expectation for a MAS is no gain
• Gains need system structure that matches the task structure
• Verification is the lever on error amplification, and in automatic MAS
it is usually broken
Paper 2: capability ceiling
single agent above ~45% → more agents hurt
• Evaluate with cost in view
Paper 3: competency floor
weak base model → MAS cannot help, strong model → it can
What is left unexplained paper 2 matches token budgets, paper 3 does not. Whether budget matching alone flips the sign has not been tested
directly
20


# Page. 21

![Page Image](https://bcdn.docswell.com/page/VJP39WXWE8.jpg)

Summary: three questions, three answers
1 Do they work?
2 Which structures help?
3 Do they beat the single agent?
Cemri et al. 2025
Kim et al. 2026
Jwalapuram et al. 2026
• 41% to 87% of runs fail across seven
frameworks
• It depends on the task: +81% (parallel financial
analysis) to −70% (sequential crafting)
• Automatic MAS: no. They collapse to sampling
at ~10× the cost
• Failures are mostly organizational: design
44%, inter-agent misalignment 32%, verification
24%
• Changing the topology helped more than
changing the prompt, in one of two tests
• Centralized verification contains error
amplification, independent agents do not
• Deliberate MAS: yes, 57% → 96.5%, but only
on a strong model and a task with real structure
• Above ~45% single-agent accuracy, adding
agents tends to hurt
• Cost-matched comparison is what separates
coordination from sampling
Bottom line a team of identical models gets no gain by default. Gains appear when the task has parallel structure the team can split, a
verifier that actually checks, and a cost-matched comparison to prove the gain is coordination and not sampling.
21


# Page. 22

![Page Image](https://bcdn.docswell.com/page/2EV4M8L9JQ.jpg)

Where the field is heading
Cost-matched comparison as the standard
Capability measured relative to the task
Kim matches token budgets, Tran & Kiela (2604.02460) match thinking tokens,
Jwalapuram reports dollars. The field is converging on: no cost control, no claim
Ceiling (paper 2) and floor (paper 3) read as one curve once capability is the single
agent’s accuracy on that task, not model size
Verification and termination as the design levers
Collective dynamics at large N
MAST’s verification category, Kim’s 17.2× vs. 4.4×, and Jwalapuram’s verifier that
picks the first candidate. Unaware-of-termination is 12.4% of MAST failures and
has no targeted fix yet
Physics of Agents (El et al. 2608.16578, Ising-type fit on 32-agent communities),
Ecology of AI Agents (Crawley & Tanaka 2610.12436, population threshold), MultiAgent Teams Hold Experts Back (Pappu et al., ICML 2026)
My own work (M1, with Kojima san) if five copies share the same weights, where can a gain come from? One possible answer is context, not weights. Splitting
context across agents removes what a single agent fails to use from a long input (loss 1) but drops information at each hand-off (loss 2). Prediction: a MAS helps when loss 1
exceeds loss 2, so under a matched token budget the sign of MAS − SAS should flip with the single agent’s task-relative capability.
22