aiqbench analysis
This paper introduces Orthrus, a serving system that unifies embedding and generative model workloads through heterogeneous batching to resolve GPU utilization inefficiencies in modern retrieval pipelines. By optimizing batch composition dynamically rather than relying on coarse-grained partitioning, the approach significantly boosts throughput and reduces latency for iterative RAG benchmarks. The release of this code offers practical optimizations for deploying mixed AI inference tasks on current hardware architectures.
Efficient Iterative Retrieval with Heterogeneous Batching
arXiv:2609.25405v1 Announce Type: new Abstract: Modern information retrieval increasingly employs both embedding and generative models to handle complex queries. However, current serving systems suffer from low throughput and poor GPU utilization because they execute these models in isolation. Coarse-grained partitioning, such as dedicating GPUs to specific tasks, fails to adapt to dynamic workloads and creates computational "bubbles". To address these, we present Orthrus, a serving system that performs heterogeneous batching within a unified inference loop. The primary challenge lies in unifying embedding and generation workloads with conflicting computational patterns while optimizing batch composition for high performance. Orthrus addresses these challenges through chunked embedding with incremental pooling and by adjusting batch composition in a workload-aware manner. Evaluation on four A100 GPUs shows that, relative to baseline deployments, Orthrus achieves 1.28$\times$--4.52$\times$ higher throughput on controlled workloads and up to 55.8% lower end-to-end p99 latency on an iterative-RAG benchmark. We release our code at https://github.com/illinoisdata/Orthrus .
Abstract:Modern information retrieval increasingly employs both embedding and generative models to handle complex queries. However, current serving systems suffer from low throughput and poor GPU utilization because they execute these models in isolation. Coarse-grained partitioning, such as dedicating GPUs to specific tasks, fails to adapt to dynamic workloads and creates computational "bubbles". To address these, we present Orthrus, a serving system that performs heterogeneous batching within a unified inference loop. The primary challenge lies in unifying embedding and generation workloads with conflicting computational patterns while optimizing batch composition for high performance. Orthrus addresses these challenges through chunked embedding with incremental pooling and by adjusting batch composition in a workload-aware manner. Evaluation on four A100 GPUs shows that, relative to baseline deployments, Orthrus achieves 1.28$\times$--4.52$\times$ higher throughput on controlled workloads and up to 55.8% lower end-to-end p99 latency on an iterative-RAG benchmark. We release our code at this https URL .
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2609.25405 [cs.AI]
(or arXiv:2609.25405v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.25405 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 248 words
Related coverage
Pre-Training Hybrid LLMs from Scratch in Primus on AMD Instinct™ GPUs
AMD · Oct 8, 2026
Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel Optimization
Meta Engineering · Jul 15, 2026
Intelligence is Free, Now What? <br> Data Systems for, of, and by Agents
BAIR Blog · Jul 7, 2026
FunnelAL: Retrieve-then-Rank Active Learning for Single-Class Discovery
arXiv · Jul 29, 2026