VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?

Title:VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?

Abstract:For years, we have built LLM serving systems like any other critical infrastructure, with a single general-purpose stack hand-tuned over many engineer-years to support every model and workload. In this paper, we take the opposite bet and let coding agents automatically synthesize bespoke serving systems for different usage scenarios. We propose VibeServe, the first agentic loop that generates entire LLM serving stacks end-to-end. VibeServe uses an outer loop to plan and track the search over system designs, and an inner loop to implement candidates, check correctness, and measure performance on the target benchmark. In standard serving settings, VibeServe matches vLLM and SGLang on Llama-3.1-8B, and delivers 2.3x the goodput of a tuned SGLang for a 397B MoE model on four AMD MI300As. In five non-standard scenarios, VibeServe outperforms existing systems by exploiting opportunities that generic systems miss, spanning non-standard model architectures, workload knowledge, and hardware-specific optimizations. Together, these results argue that infrastructure software should favor generation-time specialization over runtime generality. Code is available at this https URL.

Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)

arXiv:2605.06068 [cs.AI]

(or arXiv:2605.06068v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2605.06068 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.