MASTEST: A LLM-Based Multi-Agent System For Testing RESTful APIs

Abstract:Testing RESTful API is increasingly complicated but indispensable to quality assurance of cloud-native applications. This paper reports a multi-agent system called MASTEST that combines LLM-based intelligent agents and programmed agents to automate REST API testing. They form a complete tool chain covering the whole workflow of REST API test with API specification in the OpenAPI Swagger format as the input. It also incorporates human testers in the process to review and correct LLM generated test artefacts to control the quality of testing activities. MASTEST is evaluated on two LLMs, GPT-4o and DeepSeek V3.1 Reasoner with five public APIs. Its performances on various testing activities are measured by a wide range of metrics, including adequacy and coverage metrics, the syntax and data type correctness of generated test scripts, the usability of LLM generated test cases and scripts, as well as the bug detection ability. Experiment results demonstrated that both DeepSeek and GPT-4o achieved a high overall performance but had strengths and weaknesses on different testing activities. MASTEST generated test cases achieved 94% and 98% unit test coverage and 79% and 78% system test coverage for GPT-4o and DeepSeek respectively in comparison with human designed test cases. The generated test scripts maintained 100% syntax correctness and only required minimal manual edits for semantic correctness. The generated test scripts contain assertions on the expected status code as well as contents in the response messages. They are highly capable of detecting bugs in the REST APIs. Experiment data shows that the bug detection rates are between 2.13 to 4.50 per operation. These findings indicate that MASTEST is highly efficient and effective.

This is a revised version of the paper. It is formally published by IEEE in the Proceedings of the 8TH IEEE International Conference on AI Test (IEEE AITest 2026). Reader are also referred to the first version for technical details

Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

arXiv:2511.18038 [cs.SE]

(or arXiv:2511.18038v2 [cs.SE] for this version)

X. Han and H. Zhu, "MASTEST: A LLM-Based Multi-Agent System For Testing RESTful APIs," 2026 IEEE International Conference on Artificial Intelligence Testing (AITest), Fukuoka, Japan, 2026, pp. 235-248

https://doi.org/10.1109/AITest70988.2026.00043. DOI(s) linking to related resources