Don't Ask: Understanding What Numeric Self-Reports Measure in Language Models

Abstract:Language models are often asked to rate their internal states with a number, but it is unclear how reliable this measure is. We train probes for four affective concepts (wellbeing, interest, focus and impulsivity) in eight instruction-tuned models to track these states over multi-turn conversations. We compare each probe's score with a numeric self-report of the same state, elicited at every turn. We find that rating questions displace the probed state, and that the rating is only weakly coupled with the probe score before the question but tracks the post-question probe much more strongly. Steering experiments suggest that models use the affective directions to inform ratings, but that relationship is not specific: ratings are associated with other concepts' probes as much as with their own, and change little when the question is about the user. Finally, neither the pre-question coupling nor concept specificity improves with scale (1B-27B). We warn about the effect a question can have on the internal states one is asking about, and caution against numeric self-report as a reliable measure of those states.

41 pages, 30 figures, 18 tables. Replaces the preprint "Quantitative Introspection in Language Models: Tracking Emotive States Across Conversation" with an extended, revised study: new title; 8 models (previously 1 in depth, 4 in less detail); 100 conversations (40 previous, 60 new); probe readouts before and after the rating question (previously only after); new controls; revised conclusions

Artificial Intelligence (cs.AI)

arXiv:2603.18893 [cs.AI]

(or arXiv:2603.18893v3 [cs.AI] for this version)