aiqbench analysis
This paper addresses the latency and privacy constraints of on-device LLM inference by proposing a collaborative framework that splits computation between edge devices and the cloud. By leveraging authenticated KV caches and encrypting transmitted data, it significantly reduces downlink payloads while maintaining model performance across heterogeneous hardware. The work represents a notable advancement in balancing computational efficiency with user data security for practical AI deployment.
Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models
arXiv:2607.13093v4 Announce Type: replace-cross Abstract: On-device LLM inference faces a trilemma of response latency, limited hardware resources and user privacy. Full cloud inference delivers strong computing power but exposes user prompts and dialogue data, while standalone on-device inference is unfeasible for most consumer and embedded edge devices. This paper presents a privacy-centric edge-cloud collaborative LLM inference framework built on endpoint-authenticated KV cache. Local endpoints handle input preprocessing, embedding computation, adaptive feature optimization, KV cache authentication, speculative decoding and low-dimensional model head calculation, while the cloud conducts authenticated decoder inference, KV cache management, token verification and high-dimensional vocabulary projection. Endpoints fuse partial outputs, apply language-adaptive masking and sample target tokens. All transmitted data and truncated logits are quantized and AES-GCM encrypted for privacy, with core lightweight modules, draft parameters and cache access policies kept local to avoid leakage. The framework supports heterogeneous devices including CPU-only, GPU-equipped and embedded devices via optimized streaming, batching and quantized ONNX deployment. Evaluations demonstrate that the framework reduces per-token latency by up to 46.1\% and downlink payloads by up to 67.4\% over baseline split inference, retaining comparable performance to full cloud inference.
Abstract:On-device LLM inference faces a trilemma of response latency, limited hardware resources and user privacy. Full cloud inference delivers strong computing power but exposes user prompts and dialogue data, while standalone on-device inference is unfeasible for most consumer and embedded edge devices. This paper presents a privacy-centric edge-cloud collaborative LLM inference framework built on endpoint-authenticated KV cache. Local endpoints handle input preprocessing, embedding computation, adaptive feature optimization, KV cache authentication, speculative decoding and low-dimensional model head calculation, while the cloud conducts authenticated decoder inference, KV cache management, token verification and high-dimensional vocabulary projection. Endpoints fuse partial outputs, apply language-adaptive masking and sample target tokens. All transmitted data and truncated logits are quantized and AES-GCM encrypted for privacy, with core lightweight modules, draft parameters and cache access policies kept local to avoid leakage. The framework supports heterogeneous devices including CPU-only, GPU-equipped and embedded devices via optimized streaming, batching and quantized ONNX deployment. Evaluations demonstrate that the framework reduces per-token latency by up to 46.1\% and downlink payloads by up to 67.4\% over baseline split inference, retaining comparable performance to full cloud inference.
Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
arXiv:2607.13093 [cs.CR]
(or arXiv:2607.13093v4 [cs.CR] for this version)
Full text under the publisher's open licence · 196 words
Related coverage
Parameter- and Bandwidth-Efficient Edge--cloud Many-to-Many Speech-to-Text Translation
arXiv · Aug 17, 2026
Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models
arXiv · Jul 20, 2026
Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference
arXiv · Sep 1, 2026
The Battery Price of edge AI: A study of the Environmental Impact of LLM Inference on Mobile Devices
arXiv · Sep 14, 2026