aiqbench analysis
This arXiv preprint proposes training language models to output label probabilities in a single forward pass, instead of generating decisions token by token. The reported gains are modest (1.5 to 1.8 points of accuracy over a baseline), though the 182-fold latency speedup versus reasoning rollout is the more practical claim. The results come from a new benchmark (DecisionVista) and the authors' own evaluations, so independent replication and the code release will matter before anyone relies on the efficiency trade-offs.
Large Decision Models: Repurposing Large Language Models as Single-Pass Decision Makers
arXiv:2610.15501v1 Announce Type: new Abstract: Large language models make decisions through parallel prefill and sequential decoding. Although effective, this approach incurs decoding costs and does not explicitly expose decision probabilities. Existing task-specific classifiers require per-task training and offer limited generalization to unseen decision schemas. We propose Large Decision Models (LDMs), repurposing language models to return explicit probabilities over labels specified at run time using one prefill forward pass. We first adapt models to this interface through Diverse Decision Supervision (DDS), covering varied domains, rules and labels. To further narrow the performance gap to a reasoning model, Self-Posed Reasoning Distillation (SPRD) adds tasks generated by a frozen teacher, filters them for answer consistency, and trains on label distributions obtained from repeated teacher reasoning. We evaluate on DecisionVista, pairing new-schema tests with held-out human-labelled sources. Across three base models, the recipe improves accuracy over a step-matched baseline by 1.8 and 1.5 percentage points, respectively, achieving a 182-fold speedup in latency over reasoning rollout on Gemma-4-31B. Reasoning remains stronger on the new-schema test, motivating a confidence router for uncertain decisions. On development sets, routing recovers a median 92% of reasoning's accuracy advantage while generating 27-58% of its tokens. Code is available at https://github.com/JianhuiWei7/LDM.
Abstract:Large language models make decisions through parallel prefill and sequential decoding. Although effective, this approach incurs decoding costs and does not explicitly expose decision probabilities. Existing task-specific classifiers require per-task training and offer limited generalization to unseen decision schemas. We propose Large Decision Models (LDMs), repurposing language models to return explicit probabilities over labels specified at run time using one prefill forward pass. We first adapt models to this interface through Diverse Decision Supervision (DDS), covering varied domains, rules and labels. To further narrow the performance gap to a reasoning model, Self-Posed Reasoning Distillation (SPRD) adds tasks generated by a frozen teacher, filters them for answer consistency, and trains on label distributions obtained from repeated teacher reasoning. We evaluate on DecisionVista, pairing new-schema tests with held-out human-labelled sources. Across three base models, the recipe improves accuracy over a step-matched baseline by 1.8 and 1.5 percentage points, respectively, achieving a 182-fold speedup in latency over reasoning rollout on Gemma-4-31B. Reasoning remains stronger on the new-schema test, motivating a confidence router for uncertain decisions. On development sets, routing recovers a median 92% of reasoning's accuracy advantage while generating 27-58% of its tokens. Code is available at this https URL.
Artificial Intelligence (cs.AI)
arXiv:2610.15501 [cs.AI]
(or arXiv:2610.15501v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.15501 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 291 words
Related coverage
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · May 19, 2026
Frontier AI Risk Report (February–March 2026)
METR · May 19, 2026