aiqbench analysis
This survey reframes LLM efficiency by treating data, memory, and compute as an interacting system rather than isolated components, emphasizing that GPU memory is typically the dominant bottleneck in fine-tuning. It argues that optimal data subsets are regime-specific and that effective scaling requires joint optimization of weight storage, optimizer states, and activations alongside adaptive compute allocation. For practitioners, the key takeaway is shifting from universal best practices to resource-conditioned decision-making where training halts or reallocates based on diminishing marginal returns relative to finite FLOP budgets.
Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey
From arXiv
Full textarXiv:2606.10706v1 Announce Type: new Abstract: Resource constraints increasingly determine what can be trained, fine-tuned, and deployed in large language models (LLMs), yet efficiency is often studied through isolated techniques rather than as an interacting system of limits. This survey adopts a constraint-centric perspective and organizes recent progress around three coupled bottlenecks: data efficiency (what to train on), memory efficiency (how to fit training), and compute budget awareness (when and where to spend FLOPs).
On the data axis, we review selection and pruning methods that maximize learning per token, ranging from scalable proxy signals based on learning dynamics to gradient- and influence-based scoring, as well as difficulty-aware and curriculum-style strategies. We highlight emerging evidence that different notions of good data dominate in different regimes, implying that optimal subsets depend on the task objective and resource budget rather than being universal. On the systems side, we show that GPU memory, not raw compute, is often the dominant bottleneck in fine-tuning, and that effective scaling requires jointly reducing weight storage, optimizer states, and activation memory rather than optimizing any single component in isolation.
Beyond memory, we frame training and inference as compute-governed processes in which optimization, data selection, and decoding must explicitly account for finite FLOP budgets. We review evidence for compute-optimal allocation and stopping rules, where computation should be halted or reallocated once marginal performance gains fall below a budget-dependent threshold. Together, these results unify compute-aware data selection, scaling laws, and adaptive inference under a common principle of resource-conditioned decision-making.
Abstract:Resource constraints increasingly determine what can be trained, fine-tuned, and deployed in large language models (LLMs), yet efficiency is often studied through isolated techniques rather than as an interacting system of limits. This survey adopts a constraint-centric perspective and organizes recent progress around three coupled bottlenecks: data efficiency (what to train on), memory efficiency (how to fit training), and compute budget awareness (when and where to spend FLOPs). On the data axis, we review selection and pruning methods that maximize learning per token, ranging from scalable proxy signals based on learning dynamics to gradient- and influence-based scoring, as well as difficulty-aware and curriculum-style strategies.
We highlight emerging evidence that different notions of good data dominate in different regimes, implying that optimal subsets depend on the task objective and resource budget rather than being universal. On the systems side, we show that GPU memory, not raw compute, is often the dominant bottleneck in fine-tuning, and that effective scaling requires jointly reducing weight storage, optimizer states, and activation memory rather than optimizing any single component in isolation.
Beyond memory, we frame training and inference as compute-governed processes in which optimization, data selection, and decoding must explicitly account for finite FLOP budgets. We review evidence for compute-optimal allocation and stopping rules, where computation should be halted or reallocated once marginal performance gains fall below a budget-dependent threshold. Together, these results unify compute-aware data selection, scaling laws, and adaptive inference under a common principle of resource-conditioned decision-making.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.10706 [cs.LG]
(or arXiv:2606.10706v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.10706 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 340 words
Related coverage
Parameterized Stripe Attention for Efficient Video Generation
arXiv · Sep 30, 2026

Theory Unifies Classical & Quantum Statistical Efficiency Limits
Quantum Zeitgeist · Aug 4, 2026
Unifying Model-Free Efficiency and Model-Based Representations via Latent Dynamics
arXiv · Jun 5, 2026
Unified Static-Dynamic Pruning for Efficient LLM Inference
arXiv · Jul 28, 2026