aiqbench analysis
This arXiv preprint tackles a practical systems problem: on unified-memory edge devices, GPU memory traffic from LLM/VLM inference can make time-critical CPU perception and control tasks miss their deadlines. Ripple's approach, converting CPU deadline slack into GPU-byte budgets, is a scheduling technique evaluated on one platform (NVIDIA DGX Spark) with four robotic task mixes, so the results are promising but not yet broadly generalizable. For readers building embodied AI systems, the useful signal is that co-locating inference with real-time control requires explicit memory-bandwidth coordination, which current hardware often doesn't expose.
Ripple: Deadline-Aware CPU-GPU Sharing for Embodied Inference
arXiv:2611.01067v1 Announce Type: cross Abstract: Unified-memory edge devices increasingly run LLM/VLM inference alongside time-critical CPU tasks for perception and control. GPU memory traffic can overwhelm these tasks, yet platforms such as DGX Spark expose neither bandwidth counters nor direct bandwidth regulation. We present Ripple, a CPU-GPU co-scheduler that combines model-derived GPU memory volumes with CPU completion feedback to regulate interference. Ripple divides inference into pausable, layer-based units and translates each CPU job's deadline slack into a GPU-byte budget, admitting work within the budgets of all overlapping jobs. Online updates track changing interference, while deadline-aware pacing and judicious SM footprint narrowing preserve inference progress. Implemented with SGLang engine hooks and a central scheduler, we evaluate Ripple on DGX Spark with four robotic CPU task mixes and LLM/VLM inference. It is the only evaluated approach that consistently meets the 2% CPU deadline-miss target and VLM deadlines across the tested deadline settings while retaining best-effort LLM throughput.
Abstract:Unified-memory edge devices increasingly run LLM/VLM inference alongside time-critical CPU tasks for perception and control. GPU memory traffic can overwhelm these tasks, yet platforms such as DGX Spark expose neither bandwidth counters nor direct bandwidth regulation. We present Ripple, a CPU-GPU co-scheduler that combines model-derived GPU memory volumes with CPU completion feedback to regulate interference. Ripple divides inference into pausable, layer-based units and translates each CPU job's deadline slack into a GPU-byte budget, admitting work within the budgets of all overlapping jobs. Online updates track changing interference, while deadline-aware pacing and judicious SM footprint narrowing preserve inference progress. Implemented with SGLang engine hooks and a central scheduler, we evaluate Ripple on DGX Spark with four robotic CPU task mixes and LLM/VLM inference. It is the only evaluated approach that consistently meets the 2% CPU deadline-miss target and VLM deadlines across the tested deadline settings while retaining best-effort LLM throughput.
Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Operating Systems (cs.OS); Performance (cs.PF)
arXiv:2611.01067 [cs.DC]
(or arXiv:2611.01067v1 [cs.DC] for this version)
https://doi.org/10.48550/arXiv.2611.01067 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 254 words