analiza aiqbench
This research addresses the scalability bottleneck in runtime scheduling for concurrent heterogeneous AI models on shared GPUs by introducing a MeanField surrogate modeling approach. By predicting performance based on local configurations and aggregate GPU state rather than combinatorial joint interactions, the method achieves near-optimal scheduling efficiency with linear sample growth as workload complexity increases. The technique effectively mitigates resource contention without SLA violations, offering a practical solution for managing complex LLM and vision workloads in dynamic environments.
MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs
arXiv:2609.02109v1 Announce Type: cross Abstract: Deploying heterogeneous AI models concurrently on a shared GPU introduces resource contention that complicates runtime scheduling. While surrogate models avoid costly online benchmarking, their profiling requirements typically grow combinatorially with the number of co-running models, limiting scalability. We propose a MeanField surrogate that predicts per-model performance from local configuration and aggregate GPU state rather than explicitly modeling all joint interactions. Experiments on concurrent LLM and vision workloads across $N \in \{2,3,4,5,6\}$ show high predictive accuracy ($R^2 \approx 0.96$) with an empirical sample budget that grows approximately linearly in $N$, in contrast to the combinatorial cost of fully joint profiling. Integrated into a genetic algorithm scheduler, the surrogate scales to an $N=5$ problem with 78,732 feasible joint configurations, remaining within 0.10% of the exhaustive search with zero SLA violations across eight dynamic workload scenarios, while complete online GA decisions take 26 ms median, about $5\times$ faster than exhaustive surrogate search.
Abstract:Deploying heterogeneous AI models concurrently on a shared GPU introduces resource contention that complicates runtime scheduling. While surrogate models avoid costly online benchmarking, their profiling requirements typically grow combinatorially with the number of co-running models, limiting scalability. We propose a MeanField surrogate that predicts per-model performance from local configuration and aggregate GPU state rather than explicitly modeling all joint interactions. Experiments on concurrent LLM and vision workloads across $N \in \{2,3,4,5,6\}$ show high predictive accuracy ($R^2 \approx 0.96$) with an empirical sample budget that grows approximately linearly in $N$, in contrast to the combinatorial cost of fully joint profiling. Integrated into a genetic algorithm scheduler, the surrogate scales to an $N=5$ problem with 78,732 feasible joint configurations, remaining within 0.10% of the exhaustive search with zero SLA violations across eight dynamic workload scenarios, while complete online GA decisions take 26 ms median, about $5\times$ faster than exhaustive surrogate search.
4 pages, 3 figures, 2 tables. Accepted for publication in IEEE Embedded Systems Letters
Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI)
arXiv:2609.02109 [cs.DC]
(or arXiv:2609.02109v1 [cs.DC] for this version)
https://doi.org/10.48550/arXiv.2609.02109 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.1109/LES.2026.3724013 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 273 cuvinte
Subiecte conexe
News
Quantum Computing Report · 25 iul. 2026
ATOMesh: Unlocking AMD Hardware for Scalable LLM Serving
AMD · 16 iun. 2026
AI Serving Platform That Adapts to Your Model
Databricks · 10 iun. 2026
When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving
NVIDIA · 9 sept. 2026