Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models

Abstract:Multi-agent LLM pipeline systems break down the task among multiple roles for better reasoning, but are benchmarked mainly with large-scale commercial models. In this study, we investigate Parishad, a structured multi-agent system involving five roles, by deploying it on Qwen2.5-7B-Instruct, a local model, on two datasets: GSM8K (500 questions) and HumanEval (164 questions), compared with prompting directly and two-call self-refinement. The multi-agent system drops GSM8K accuracy from 75.0\% to 45.0\% with JSON data format due to the error accumulation problem. With plaintext format, the accuracy is restored to 82.0\%. A two-call self-refinement strategy (V1) can achieve 86.2\% accuracy on GSM8K, with 7.4$\times$ lower token usage. However, the same V1 implementation on HumanEval---where direct accuracy is already 96.3\%---actively destroys performance (66.5\%). A task-aware gated redesign (V2) applied to HumanEval preserves accuracy at 95.1\%. Our results demonstrate that communication format and implementation details determine outcomes more than architectural complexity, and that simpler approaches match or outperform multi-agent pipelines for local 7B model deployment. All code and data are released.

Machine Learning (cs.LG)

arXiv:2607.26922 [cs.LG]

(or arXiv:2607.26922v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2607.26922 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.