aiqbench analysis
This research addresses the latency-resource-privacy trilemma in LLM deployment by introducing a collaborative inference framework that splits workloads between edge devices and the cloud. By leveraging endpoint-authenticated KV caches and encrypting transmitted data with AES-GCM, the system significantly reduces downlink payloads while maintaining model performance without exposing sensitive user prompts. The approach offers a practical pathway for heterogeneous hardware to deploy large models securely, balancing computational efficiency against privacy constraints in real-world applications.
Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models
arXiv:2607.13093v2 Announce Type: replace-cross Abstract: On-device LLM inference faces a trilemma of response latency, limited hardware resources and user privacy. Full cloud inference delivers strong computing power but exposes user prompts and dialogue data, while standalone on-device inference is unfeasible for most consumer and embedded edge devices. This paper presents a privacy-centric edge-cloud collaborative LLM inference framework built on endpoint-authenticated KV cache. Local endpoints handle input preprocessing, embedding computation, adaptive feature optimization, KV cache authentication, speculative decoding and low-dimensional model head calculation, while the cloud conducts authenticated decoder inference, KV cache management, token verification and high-dimensional vocabulary projection. Endpoints fuse partial outputs, apply language-adaptive masking and sample target tokens. All transmitted data and truncated logits are quantized and AES-GCM encrypted for privacy, with core lightweight modules, draft parameters and cache access policies kept local to avoid leakage. The framework supports heterogeneous devices including CPU-only, GPU-equipped and embedded devices via optimized streaming, batching and quantized ONNX deployment. Evaluations demonstrate that the framework reduces per-token latency by up to 46.1\% and downlink payloads by up to 67.4\% over baseline split inference, retaining comparable performance to full cloud inference.
Abstract:On-device LLM inference faces a trilemma of response latency, limited hardware resources and user privacy. Full cloud inference delivers strong computing power but exposes user prompts and dialogue data, while standalone on-device inference is unfeasible for most consumer and embedded edge devices. This paper presents a privacy-centric edge-cloud collaborative LLM inference framework built on endpoint-authenticated KV cache. Local endpoints handle input preprocessing, embedding computation, adaptive feature optimization, KV cache authentication, speculative decoding and low-dimensional model head calculation, while the cloud conducts authenticated decoder inference, KV cache management, token verification and high-dimensional vocabulary projection. Endpoints fuse partial outputs, apply language-adaptive masking and sample target tokens. All transmitted data and truncated logits are quantized and AES-GCM encrypted for privacy, with core lightweight modules, draft parameters and cache access policies kept local to avoid leakage. The framework supports heterogeneous devices including CPU-only, GPU-equipped and embedded devices via optimized streaming, batching and quantized ONNX deployment. Evaluations demonstrate that the framework reduces per-token latency by up to 46.1\% and downlink payloads by up to 67.4\% over baseline split inference, retaining comparable performance to full cloud inference.
Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
arXiv:2607.13093 [cs.CR]
(or arXiv:2607.13093v2 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2607.13093 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 277 words
Related coverage
Parameter- and Bandwidth-Efficient Edge--cloud Many-to-Many Speech-to-Text Translation
arXiv · Aug 17, 2026
Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference
arXiv · Sep 1, 2026
The Battery Price of edge AI: A study of the Environmental Impact of LLM Inference on Mobile Devices
arXiv · Sep 14, 2026
How to Build In-Vehicle AI Agents with NVIDIA: From Cloud to Car
NVIDIA · May 5, 2026