aiqbench analysis
This research demonstrates that apparent reasoning failures in large language models often stem from output bottlenecks rather than a lack of internal logic, as hidden-state probes can recover correct answers even when sequence scoring collapses. By introducing a minimal, label-free calibration protocol, the study successfully restores significant accuracy across diverse benchmarks like ProofWriter and FOLIO without requiring extensive retraining or target labels. These findings urge a more nuanced interpretation of zero-shot reasoning deficits, suggesting that many models possess intact internal logic that is merely masked by expression failures.
Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
arXiv:2608.31068v1 Announce Type: new Abstract: When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of reasoning with a late-stage output bottleneck. We observe a consistent readout gap across diverse reasoning benchmarks: hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses due to structural biases. To test whether instance-specific logic survives this collapse, we introduce a diagnostic protocol using a minimal, target-label-free additive correction. Fitting just two parameters on as few as 25 unlabeled examples recovers 9--34 accuracy points for Qwen3.5 models, transferring successfully to OLMo-2-1B and Llama-3.1-8B. Crucially, these recovered decisions persist on hard instances unresolved by simple lexical overlap and significantly exceed count-preserving permutation baselines. Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic, urging a narrower interpretation of benchmark evaluations.
Abstract:When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of reasoning with a late-stage output bottleneck. We observe a consistent readout gap across diverse reasoning benchmarks: hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses due to structural biases. To test whether instance-specific logic survives this collapse, we introduce a diagnostic protocol using a minimal, target-label-free additive correction. Fitting just two parameters on as few as 25 unlabeled examples recovers 9--34 accuracy points for Qwen3.5 models, transferring successfully to OLMo-2-1B and Llama-3.1-8B. Crucially, these recovered decisions persist on hard instances unresolved by simple lexical overlap and significantly exceed count-preserving permutation baselines. Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic, urging a narrower interpretation of benchmark evaluations.
20 pages, 4 figures, and 43 appendix tables. Research paper on language-model interpretability, reasoning evaluation, output-scoring bottlenecks, and label-free calibration. The paper evaluates controlled three-way logical reasoning tasks, ProofWriter, ANLI, and FOLIO using Qwen3.5, OLMo-2-1B, Llama-3.1-8B, and Pythia checkpoints
Artificial Intelligence (cs.AI)
arXiv:2608.31068 [cs.AI]
(or arXiv:2608.31068v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.31068 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 200 words
Related coverage
Lattice-Based Cryptography Explained
Quantum Zeitgeist · Aug 15, 2026
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026
How to build a cancer vaccine, and whether they will work this time
LessWrong · Jun 8, 2026
Handling Imbalanced Classification: What Works Better Than SMOTE
Analytics Vidhya · Jul 12, 2026