ServerlessLoRA: Enabling Low-Latency Serverless Multi-LoRA Serving

Abstract:Multi-LoRA (Low-Rank Adaptation) serving allows many specialized LLM variants to share the same base model by attaching lightweight adapters. This makes it attractive for serving large catalogs of domain-, tenant-, and task-specific models. However, existing deployments waste resources on rarely used adapters, while directly running LoRA inference on serverless platforms introduces high memory overhead, repeated cold starts, and poor scheduling decisions. This paper presents ServerlessLoRA, a serverless multi-LoRA serving system that separates serving state into shared backbone state, reusable variant warm state, and request-private state. ServerlessLoRA loads each backbone once, shares it read-only across isolated LoRA functions, batches selected backbone operations, selectively warms variant state, and routes requests based on state locality and GPU contention. Evaluated on industrial traces, ServerlessLoRA reduces time-to-first-token by up to 92.3% against serverless baselines and achieves 1.66-3.01 times and 2.20-3.26 times higher latency-cost efficiency than vLLM-LoRA and dLoRA, respectively.

Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)

arXiv:2505.14468 [cs.LG]

(or arXiv:2505.14468v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2505.14468 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.