analiza aiqbench
The paper addresses a practical bottleneck: general-purpose streaming systems handle GPU-heavy LLM inference poorly. Its contributions are autoscaling across heterogeneous worker pools and prefix-aware batching, which together report 1.5x to 2x higher throughput at similar latency. These are benchmark results from a Google Cloud Dataflow system, not independent validation, so readers running LLMs in streaming pipelines should test the gains on their own workloads.
Accelerated Dataflow: scaling inference in streaming data analytics with efficiency
arXiv:2610.22731v1 Announce Type: cross Abstract: Streaming data analytics helps businesses identify opportunities and spot risks in a timely fashion. As large language model (LLM) usage grows, people are exploring incorporating agents or LLMs in streaming data pipelines to help find patterns in datasets without supervision or interpret natural languages for content moderation. However, most streaming systems are currently optimized for data movement and perform poorly on GPU intensive workloads such as performing inference with LLMs. We introduce a system on Google Cloud Dataflow designed to tackle hybrid streaming machine learning workloads with co-designed autoscaling on heterogeneous worker pools and prefix-aware batching. This increases throughput 1.5x to 2x for streaming pipelines using LLMs while maintaining similar latency with up to 75% peak resource savings and 84% lower peak tail latency compared with existing streaming data processing systems.
Abstract:Streaming data analytics helps businesses identify opportunities and spot risks in a timely fashion. As large language model (LLM) usage grows, people are exploring incorporating agents or LLMs in streaming data pipelines to help find patterns in datasets without supervision or interpret natural languages for content moderation. However, most streaming systems are currently optimized for data movement and perform poorly on GPU intensive workloads such as performing inference with LLMs. We introduce a system on Google Cloud Dataflow designed to tackle hybrid streaming machine learning workloads with co-designed autoscaling on heterogeneous worker pools and prefix-aware batching. This increases throughput 1.5x to 2x for streaming pipelines using LLMs while maintaining similar latency with up to 75% peak resource savings and 84% lower peak tail latency compared with existing streaming data processing systems.
Distributed, Parallel, and Cluster Computing (cs.DC); Databases (cs.DB); Machine Learning (cs.LG)
arXiv:2610.22731 [cs.DC]
(or arXiv:2610.22731v1 [cs.DC] for this version)
https://doi.org/10.48550/arXiv.2610.22731 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 231 cuvinte
Subiecte conexe
DxPTA: An Architecture Design Space Exploration with Optical Dataflow-guided Strategy for HW/SW Co-Design of Photonic Transformer Accelerators
arXiv · 8 iun. 2026
Algebraic Multigrid Acceleration for Efficient Label Spreading
arXiv · 28 aug. 2026
Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators
arXiv · 22 iul. 2026
Error-Based Bidirectional Predictive Coding Enables Deep, 4-Bit Predictive Coding Networks
arXiv · 14 oct. 2026