【DL輪読会】“Multi-Agent LLM Systems: Do They Work, Which Structures Help, and Do They Beat a Single Agent?”

DEEP LEARNING JP [DL Papers] “Multi-Agent LLM Systems: Do They Work, Which Structures Help, and Do They Beat a Single Agent?” Presenter: Tsogbadrakh Khangai, Matsuo-Iwasawa Lab M1 http://deeplearning.jp/ 1

• http://deeplearning.jp/

Background: Multi-agent LLM systems Single agent (SAS) Multi-agent (MAS) Planner LLM Coder think, act, observe, repeat Reviewer messages, feedback loops tools / env tools / env Three mechanisms a team is supposed to add Decomposition split a big task into parts Specialization each role does what it is good at Verification another agent checks the output 2

Three MAS papers Paper 1 Question What they study Do they work, and when they fail, why Hand-built open-source frameworks (MetaGPT, ChatDev, AG2 ...) Kim et al., Google Research / DeepMind / MIT. Published as “Capable language models can outgrow the benefits of collaboration”, Nature Machine Intelligence 2026 Which structures are actually good Five canonical topologies built by the authors, matched token budgets The Illusion of Multi-Agent Advantage Even with a good structure, does it beat the single agent you already have Automatically generated MAS (ADAS, AFlow, DyLAN ...) vs. self-consistency Why Do Multi-Agent LLM Systems Fail? Cemri et al., UC Berkeley, NeurIPS 2025 Towards a Science of Scaling Agent Systems 2 3 Jwalapuram et al., Salesforce Research, NeurIPS 2026 3

MAS failures, and what they are tied to PAPER 1: WHY DO MULTI-AGENT LLM SYSTEMS FAIL? 4

Setup: 1,642 traces, 7 frameworks Data MAST-Data: 1,642 annotated execution traces, 7 frameworks (MetaGPT, ChatDev, HyperAgent, AppWorld, AG2, Magentic-One, OpenManus) Models GPT-4 / GPT-4o, Claude 3.7 Sonnet, Qwen2.5-Coder-32B, CodeLlama-7B Taxonomy Grounded Theory on 150 traces, 6 expert annotators, human agreement κ = 0.88 Scaling LLM-as-judge with o1 (few-shot): 94% accuracy, κ = 0.77 vs. humans, ~$1.8 per trace Link arxiv.org/abs/2503.13657 Benchmarks in plain words ProgramDev write a small program from a spec AppWorld operate phone apps through APIs HumanEval short Python functions OlympiadBench olympiad math problems SWE-Bench Lite fix real GitHub issues GAIA assistant questions that need web and tools 5

Results: MAS failed in 41% to 86.7% of runs 0 AG2 (OlympiadBench) MetaGPT (ProgramDev) 10 20 30 40 50 60 70 80 90 41.0% 100 Analysis • Each system is on a different benchmark, so this is not a ranking • Models: GPT-4o and Claude 3.7 Sonnet 60.0% • Failure = task not completed as specified • Harder benchmarks had higher failure rates Magentic-One (GAIA) ChatDev (ProgramDev) HyperAgent (SWE-Bench Lite) AppWorld (Test-C) 62.0% 66.7% 74.7% 86.7% Figure 5 of the paper 6

Results: MAST 14 failure modes in 3 categories Category FC1 System design issues 44.2% of failures FC2 Inter-agent misalignment 32.3% of failures FC3 Task verification 23.5% of failures Failure mode Share 1.1 Disobey task specification 11.8% 1.2 Disobey role specification 1.5% 1.3 Step repetition 15.7% 1.4 Loss of conversation history 2.8% 1.5 Unaware of termination conditions 12.4% 2.1 Conversation reset 2.2% 2.2 Fail to ask for clarification 6.8% 2.3 Task derailment 7.4% 2.4 Information withholding 0.8% 2.5 Ignored other agent’s input 1.9% 2.6 Reasoning-action mismatch 13.2% 3.1 Premature termination 6.2% 3.2 No or incomplete verification 8.2% 3.3 Incorrect verification 9.1% The three largest • Step repetition 15.7% • Reasoning-action mismatch 13.2% • Unaware of termination 12.4% Shares are percentages of all observed failures across 1,642 traces, not failure rates. Figure 1 of the paper 7

Results: Both model and architecture matter Model matters Architecture matters • Inside MetaGPT, GPT-4o has 39% fewer system-design failures than Claude 3.7 Sonnet • Same task: MetaGPT has 60 to 68% fewer FC1 / FC2 failures than ChatDev, but 1.56× more FC3 • Qwen2.5-Coder-32B is far more robust than CodeLlama-7B • Failures shift between categories rather than disappear Interventions guided by MAST (task success, %) ChatDev ProgramDev-v0 ChatDev HumanEval AG2 GSM-Plus (GPT-4) AG2 GSM-Plus (GPT-4o) Baseline 25.0 89.6 84.75 84.25 Improved prompt 34.4 90.3 89.75 89.00 New topology 40.6 91.5 85.50 88.83 8

Doubts • Topology change works in one of two experiments AG2 with GPT-4 shows no significant gain (p = 0.4) and ties with prompt tuning on GPT-4o • Taxonomy comes from 150 traces, five frameworks, 2024-era models whether the mode frequencies hold for 2026 models is untested, as the authors note • Fixes reduce failures, they do not make MAS good ProgramDev completion after the best intervention is still 40.6% Handoff failures are mostly organizational, and changing the topology helped more than changing the prompt. So which topologies are worth building? 9

Which architectures are actually good? PAPER 2: TOWARDS A SCIENCE OF SCALING AGENT SYSTEMS 10

Setup: 5 architectures, 6 tasks Single agent Independent A A A Centralized Hybrid A O O A A one loop Decentralized parallel, no talking A everyone debates A A A orchestrator + workers A A A orchestrator + side talk Models GPT-5 nano / mini / GPT-5, Gemini 2.0 Flash / 2.5 Flash / 2.5 Pro, Claude Sonnet 3.7 / 4 / 4.5 Control 260 configurations, same tools and prompts, matched maximum token budget (team shares one budget, mean ~4,800 reasoning tokens per trial) Link arxiv.org/abs/2512.08296 Benchmarks in plain words BrowseComp-Plus find obscure facts by browsing the web Workbench office tasks with 16 tools (email, calendar ...) Finance-Agent analysis questions over financial filings SWE-bench Verified fix real GitHub issues PlanCraft Minecraft item crafting from an inventory Terminal-Bench command-line tasks 11

Results: best architecture depends on the task Benchmark SAS mean Best MAS vs. SAS Worst MAS vs. SAS Why (authors’ trace reading) Finance-Agent 0.349 Centralized +80.8% Independent +57% parallel information streams BrowseComp-Plus 0.318 Decentralized +9.2% Independent −35% high-entropy search, needs cross-checks Workbench 0.629 Decentralized +5.6% −11% 16 tools, coordination overhead Terminal-Bench 0.344 Independent +1.7% Centralized −19.2% 2 tools, orchestration not justified SWE-bench Verified 0.522 Hybrid −2.1% Independent −14.9% strong single-agent baseline PlanCraft 0.568 Hybrid −39.1% Independent −70.0% sequential 3-step path, no parallelism −0.3% mean change across all MAS [−58.7%, +77.2%] Gains when the task splits into parallel streams Losses when the task is a short sequential path and coordination only adds turns 95% confidence interval Relative change = (mean MAS − mean SAS) / mean SAS. Figure 2 and Section 4.1 of the paper 12

Results: Capability ceiling exists ~45% 17.2× → 4.4× Tools < 5 Capability ceiling Error amplification Tool-coordination trade-off Above ~45% single-agent accuracy, adding agents tends to hurt The only regularity that survives multiplecomparison correction Independent agents amplify trace errors 17.2×, centralized verification holds it to 4.4×. Aggregation without a check lets errors through More tools, larger coordination penalty. Tasks with four or fewer tools show little effect Predictive model • Picks the best architecture for 87% of held-out configurations (random: 20%, capability-only: 54%) • Cross-validated R² = 0.37, so it ranks architectures but does not predict absolute scores across domains 13

Doubts • Budget matching cuts both ways fair on cost, but each agent in a 5-agent team gets a fifth of the tokens. The capability ceiling may partly be a starvation effect on hard tasks • Six benchmarks means six clusters dataset-level claims are called “directional” by the authors themselves • R² = 0.37 is a weak model trust the 87% architecture-selection number, not the absolute predictions Handoff a hand-designed team wins on some tasks. But most people will not hand-design one. They will use a framework that builds the MAS for them, or just sample one model a few times. How does MAS compare to that? 14

Does MAS beat the single agent you already have? PAPER 3: THE ILLUSION OF MULTIAGENT ADVANTAGE 15

Setup: automatic MAS vs. CoT-SC Single-agent baseline CoT and CoT-SC (ask once, sample 5, majority vote). CoT-SC is the main comparison Automatic MAS Inference-time: DyLAN (top-K agent selection), MAS-Zero (meta-agent + verifier) Pre-optimized: ADAS, AFlow (workflow search), MaAS (supernet sampling), MAS-Orchestra (RL-trained orchestrator) Backbones GPT-4o, GPT-5, GPT-OSS-120B, Gemini 2.5 Pro. Cost reported in dollars, not matched Link arxiv.org/abs/2606.13003 Benchmarks in plain words GPQA-Diamond PhD-level science, multiple choice BrowseComp-Plus find obscure facts by browsing HLE-Maths Humanity’s Last Exam, math subset SMFR (new) synthetic multi-hop financial reasoning over per-investor transaction records, built to have MAS-friendly structure SWE-Bench Lite fix real GitHub issues 16

Results: sampling, not coordination Opening the box GPT-5 Accuracy Cost ($) GPQA-Diamond CoT-SC 87.35% 46 • DyLAN agents vote unanimously in >90% of GPT-5 cases. Expert roles make no difference (53.4 vs. 54.4 generic) GPQA-Diamond ADAS 85.23% 832 BrowseComp-Plus CoT-SC 83.92% 67 • MAS-Zero verifier picks the first candidate >45% of the time. Later workers are “expensive witnesses” BrowseComp-Plus MAS-Zero 71.63% 1,370 Same backbone, same benchmark: CoT-SC matches or beats every automatic MAS on the standard benchmarks, at roughly 10× lower cost • AFlow 7 of 14 final workflows are repeat-one-prompt-and-aggregate • MaAS 74% of BrowseComp-Plus calls collapse to a single I/O step • MAS-Orchestra picks Debate agents 85% of the time on GPQA regardless of difficulty The committee is five clones of one person No decomposition, no specialization, just more samples Exception the authors state: on HLE-Maths, GPT-5 and Gemini 2.5 Pro get significant MAS gains (GPT-5 CoT-SC 33.9% vs. MAS-Zero 38.2%). Table 4 and Section 5 of the paper 17

Results: A deliberately designed MAS still wins Expert-MAS on SMFR, GPT-5 Same design, other backbones 57.0% 96.5% CoT-SC Expert-MAS • Cost comparable: $555 vs. $478 for CoT-SC • Hand-built: meta-agent parses the problem, extractor and calculator agents handle sub-tasks, deterministic Python code orchestrates Backbone CoT-SC Expert-MAS GPT-5 57.0% 96.5% GPT-OSS-120B 26.1% 36.1% GPT-4o 22.1% 22.6% On GPT-4o the structure cannot rescue calculation and retrieval failures. The authors call this a competency floor • Strict context separation, no “orchestration noise” The paper’s own conclusion automatic MAS is an illusion, deliberate MAS needs a strong enough base model and a task with real structure 18

Doubts • The authors built both the benchmark and the system that aces it SMFR and Expert-MAS were designed together. 96.5% is a proof of existence, not a general result • Dollar cost mixes architecture cost with vendor pricing token counts would separate the two • Tool parity is not stated Expert-MAS runs a deterministic Python executor and calculator agents. Whether CoT-SC had the same tools is not in the paper. If not, part of 57 → 96.5 is tools, not coordination Consistent with paper 2 under a matched budget, paper 2 shows the sign of the MAS effect depends on task structure. Without cost matching, paper 3 shows the sign looks positive for the wrong reason: sampling volume. Together, a MAS comparison without cost control proves nothing 19

Summary: agreement, and one open question They agree They seem to contradict • The default expectation for a MAS is no gain • Gains need system structure that matches the task structure • Verification is the lever on error amplification, and in automatic MAS it is usually broken Paper 2: capability ceiling single agent above ~45% → more agents hurt • Evaluate with cost in view Paper 3: competency floor weak base model → MAS cannot help, strong model → it can What is left unexplained paper 2 matches token budgets, paper 3 does not. Whether budget matching alone flips the sign has not been tested directly 20

Summary: three questions, three answers 1 Do they work? 2 Which structures help? 3 Do they beat the single agent? Cemri et al. 2025 Kim et al. 2026 Jwalapuram et al. 2026 • 41% to 87% of runs fail across seven frameworks • It depends on the task: +81% (parallel financial analysis) to −70% (sequential crafting) • Automatic MAS: no. They collapse to sampling at ~10× the cost • Failures are mostly organizational: design 44%, inter-agent misalignment 32%, verification 24% • Changing the topology helped more than changing the prompt, in one of two tests • Centralized verification contains error amplification, independent agents do not • Deliberate MAS: yes, 57% → 96.5%, but only on a strong model and a task with real structure • Above ~45% single-agent accuracy, adding agents tends to hurt • Cost-matched comparison is what separates coordination from sampling Bottom line a team of identical models gets no gain by default. Gains appear when the task has parallel structure the team can split, a verifier that actually checks, and a cost-matched comparison to prove the gain is coordination and not sampling. 21

Where the field is heading Cost-matched comparison as the standard Capability measured relative to the task Kim matches token budgets, Tran & Kiela (2604.02460) match thinking tokens, Jwalapuram reports dollars. The field is converging on: no cost control, no claim Ceiling (paper 2) and floor (paper 3) read as one curve once capability is the single agent’s accuracy on that task, not model size Verification and termination as the design levers Collective dynamics at large N MAST’s verification category, Kim’s 17.2× vs. 4.4×, and Jwalapuram’s verifier that picks the first candidate. Unaware-of-termination is 12.4% of MAST failures and has no targeted fix yet Physics of Agents (El et al. 2608.16578, Ising-type fit on 32-agent communities), Ecology of AI Agents (Crawley & Tanaka 2610.12436, population threshold), MultiAgent Teams Hold Experts Back (Pappu et al., ICML 2026) My own work (M1, with Kojima san) if five copies share the same weights, where can a gain come from? One possible answer is context, not weights. Splitting context across agents removes what a single agent fails to use from a long input (loss 1) but drops information at each hand-off (loss 2). Prediction: a MAS helps when loss 1 exceeds loss 2, so under a matched token budget the sign of MAS − SAS should flip with the single agent’s task-relative capability. 22