Scoring the Wrong Question: Readout Failures in Constrained-Option Evaluation

Abstract:Constrained-option scoring reads a model's probabilities for a fixed set of permitted answers, so it returns a score even when the model is about to write something else. We study a prompt that quotes a multiple-choice item, ending in the item's own answer instruction, and asks for a forecast of whether a reader model given only a short note will answer it correctly. The model can begin answering the quoted item: on three Qwen3 checkpoints an answer letter is the most probable token on nearly every passage, and the forecast ranks the reader's correctness indistinguishably from chance. Beyond prior work's argmax check and prefill repair, we contribute a label-free diagnostic of the declared options' support, their first-token mass in context; a deletion test tracing the Qwen3 answer-letter start to the quoted instruction; and rendering and tokenisation checks for dropped separators and coinciding first tokens. Prefilling an answer stem lifts the median option mass from below to above one half on all nine checkpoints scored with this prompt and, with options and renormalisation unchanged, raises the Qwen3-averaged AUC from 0.4929 to 0.6152. lm-polygraph's default P(True) estimator shows a related failure: it reads True where Qwen3 without thinking favours an answer letter (MMLU) or the prompt's (A)/(B) label (TriviaQA), and its Qwen3-averaged AUROC is indistinguishable from chance on MMLU and below chance on TriviaQA. Scoring and renormalising the (A)/(B) labels after an answer stem raises it to 0.632 and 0.868, renormalising True against False in place reaches 0.660 and 0.860, and on TriviaQA both exceed all fourteen default single-answer library estimators. The original forecast is renormalised too, yet stays indistinguishable from chance: option mass flags low support in both settings without labels, and only checking the intended target shows which score still ranks.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

arXiv:2609.33179 [cs.CL]

(or arXiv:2609.33179v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2609.33179 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

supplementary/MANIFEST.json

supplementary/README.md

supplementary/crossfamily_table3.json

supplementary/forecast_five_readouts.json

supplementary/forecast_two_positions.json

supplementary/labels_hand_first.json

supplementary/labels_hand_second.json

supplementary/labels_judges.json

supplementary/lmpolygraph_mmlu500.json

supplementary/lmpolygraph_mmlu500_wrong_printed_answer.json

supplementary/lmpolygraph_mmlu64.json

supplementary/lmpolygraph_triviaqa500.json

supplementary/lmpolygraph_triviaqa500_estimators.json

(8 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.