analiza aiqbench
This research addresses the critical deployment challenge of "acquisition collapse," where an LLM's aggregate performance masks its inability to provide reliable pre-call signals for selective routing. The proposed RAISE framework reframes expensive inference as an information-acquisition problem, offering a diagnostic method to distinguish between predictable incremental benefits and misleading average lifts before committing computational resources. This work is essential for optimizing cost-efficiency in real-world systems that rely on costly, on-demand large language models.
RAISE: Diagnosing Acquisition Collapse in Costly LLM Signals
arXiv:2608.10441v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used as costly, on-demand components in real systems, but calling them indiscriminately can waste substantial compute, latency, and serving budget. The key deployment question is therefore not only whether an LLM helps on average, but when it is worth calling. We identify a common failure mode, which we call acquisition collapse: an LLM signal can appear useful in aggregate or post hoc, yet still provide too little before-call information to support reliable selective use. We introduce RAISE (Reward-SNR Actionability in Signal Evaluation), a pre-routing diagnostic framework for testing whether available evidence supports selective use before committing to a routing strategy. We instantiate RAISE with Structured Hypothesis Embeddings (SHE), a frozen-LLM intent signal for recommendation using one LLM call per user, and evaluate it through controlled, retrospective, and fresh-cohort studies and a prospective offline pilot whose audit decisions are frozen before independent outcomes are revealed. Across these settings, predictable incremental benefit, not average lift alone, distinguishes settings with recoverable selective value; deployment additionally depends on cost and operational constraints. Seemingly strong oracle or subgroup gains can disappear under independent evaluation. More broadly, RAISE reframes costly inference as an information-acquisition problem: before paying for an expensive model, tool, sensor, or measurement, first test whether its value is predictable at decision time. This principle motivates cost-aware acquisition in settings ranging from agent tool use and stronger-model consultation to robotic sensing and clinical decision pipelines.
Abstract:Large language models (LLMs) are increasingly used as costly, on-demand components in real systems, but calling them indiscriminately can waste substantial compute, latency, and serving budget. The key deployment question is therefore not only whether an LLM helps on average, but when it is worth calling. We identify a common failure mode, which we call acquisition collapse: an LLM signal can appear useful in aggregate or post hoc, yet still provide too little before-call information to support reliable selective use. We introduce RAISE (Reward-SNR Actionability in Signal Evaluation), a pre-routing diagnostic framework for testing whether available evidence supports selective use before committing to a routing strategy. We instantiate RAISE with Structured Hypothesis Embeddings (SHE), a frozen-LLM intent signal for recommendation using one LLM call per user, and evaluate it through controlled, retrospective, and fresh-cohort studies and a prospective offline pilot whose audit decisions are frozen before independent outcomes are revealed. Across these settings, predictable incremental benefit, not average lift alone, distinguishes settings with recoverable selective value; deployment additionally depends on cost and operational constraints. Seemingly strong oracle or subgroup gains can disappear under independent evaluation. More broadly, RAISE reframes costly inference as an information-acquisition problem: before paying for an expensive model, tool, sensor, or measurement, first test whether its value is predictable at decision time. This principle motivates cost-aware acquisition in settings ranging from agent tool use and stronger-model consultation to robotic sensing and clinical decision pipelines.
33 pages, 12 figures. v2: substantially revised and retitled (v1 title: "Detecting an Effect Is Not Learning to Act on It: A Reward-SNR Floor for LLM Acquisition Agents"); adds the RAISE audit, a controlled mechanism study, a fresh-cohort study, and a prospective offline pilot; new coauthors
Machine Learning (cs.LG); Computation and Language (cs.CL); Information Retrieval (cs.IR)
arXiv:2608.10441 [cs.LG]
(or arXiv:2608.10441v2 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 301 cuvinte
Subiecte conexe
Inside D-Wave’s Dual-Rail Qubit Roadmap and the New Haven Bet on Logical Qubits
Quantum Zeitgeist · 11 oct. 2026
Stock Market Today Highlights, Sept 2: Sensex ends 373 pts lower at 76, 570, Nifty below 24, 000; Asian Paints, HDFC Bank fall most
thehindubusinessline.com · 2 sept. 2026
LiveLaw Supreme Court Half-Yearly Digest: January - June 2026 [Part 2]
livelaw.in · 13 sept. 2026
[AINews] OpenAI shuts off Cursor
Latent Space · 29 aug. 2026