aiqbench analysis
This research introduces a novel 2.7-bit quantization framework specifically designed to enable efficient inference of large vision-language models on resource-constrained mobile hardware like Arm CPUs. By leveraging the model itself to generate training data without access to the original setup, this method successfully compresses an 11B parameter model while maintaining strong performance on visual question answering tasks. The work addresses a critical bottleneck in deploying advanced AI capabilities on edge devices by significantly reducing memory footprint and computational overhead.
Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs
arXiv:2608.21134v1 Announce Type: cross Abstract: Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for efficient inference on resource-constrained hardware. Our approach combines a quantization pipeline that uses the model itself to generate training data and does not require access to the training setup, with a novel 2.7-bit-per-parameter format supporting efficient execution on Arm CPUs. We validate our approach by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB with 8-bit activations, preserving strong performance on a set of standard visual question answering tasks.
Title:Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs
Abstract:Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for efficient inference on resource-constrained hardware. Our approach combines a quantization pipeline that uses the model itself to generate training data and does not require access to the training setup, with a novel 2.7-bit-per-parameter format supporting efficient execution on Arm CPUs. We validate our approach by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB with 8-bit activations, preserving strong performance on a set of standard visual question answering tasks.
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2608.21134 [cs.CV]
(or arXiv:2608.21134v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2608.21134 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 202 words
Related coverage
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026
Unsloth vs Axolotl vs TRL vs LLaMA-Factory: A Fine-Tuning Framework Comparison on Speed, VRAM, and Multi-GPU
MarkTechPost · Jul 22, 2026
How to Size GPUs for AI Inference and TCO Without Overspending
NVIDIA · Sep 1, 2026