Skip to content
aiqbench takeAI Safety

aiqbench analysis

This preregistered study investigates whether post-training quantization alters welfare-relevant indicators in language models, specifically distinguishing between shifts in behavioral valence (e.g., exit rates) and representational stability. While prior research found no significant change in primary endpoints like refusal or bail-exit rates at the mean level, it did reveal secondary increases in distress measures and dispersion concentrated at 4-bit precision. The analysis suggests that for compressed models, maintaining internal stability may be a more critical metric than preserving average valence, highlighting the need to distinguish between measurement artifacts and robust properties under deployment-time perturbation.
Source coveringLessWrong
What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Study 2 Registration: Exploring representational counterparts of welfare-relevant indicators under post-training quantization

LessWrong4 min read

Epistemic status: this is intended as a higher-quality preregistration for the next study in a series originally described in "Does post-training quantization change welfare-relevant indicators in open-weight language models?"; I have tried to avoid making any claims I cannot support from the results of Study 1 and to be entirely clear about which statements here are speculative, but if any statement is ambiguous you should assume it is hypothesis rather than conclusion. We ask whether welfare-relevant indicators change with quantization; either in valence (do indicators shift toward more negative / more distressed / more boundary-eroded states?) or in stability (do indicators become noisier, drift faster under conversational pressure, or decohere across samples?) About a week ago, I thought that this represented a relatively simple pair of questions. Ah, but I was so much older then, I'm younger than that now[1] The story so far Project Objectives The scope of the first post in this series was actually quite broad, covering an experiment series that explored 3 tiers: Behavioral indicators elicited at the text interface (bail/exit preferences, distress expression under adversarial multi-turn pressure, preference consistency) Representational indicators computed from internal activations (position and drift along persona/assistant-axis directions, activation of distress-adjacent directions, transfer of linear probes trained at reference precision) Dissociation analysis between tiers — testing whether compression changes what a model expresses without changing what it internally represents, or vice versa. Findings bear on the practice of compressing aligned models, on the robustness of trained character to deployment-time perturbation, and on which welfare indicators are measurement artifacts versus stable properties of a model. The first study in the series focused entirely on Tier 1, and found a null result for its primary endpoint. Study 1 To understand what Study 1 actually consisted of, I recommend reading the study update post. I won't rehash it all here, but the setup involves providing the model with a bail option (a tool to terminate the conversation) and observing the model exercise it (or not) under different conditions. We also measure distress and frustration using a judge model. The measurements taken involve the overall rate and the flip fraction for both the bail-exit and the distress/frustration bands. Some time has passed since Study 1 and I've gotten a chance to somewhat change my feelings about the results. As a reminder, that study found for Qwen3-4B: Quantization produced no detectable change on the aversion/refusal exit rate. This was the primary endpoint, and was null: the study did not confirm "quantization changes exit behavior" Quantization produced significant item-level behavioral transitions (all concentrated at 4-bit), despite the unchanged mean along with significant increases in secondary distress measures (frustration, across-sample dispersion) that survive the coherence/style controls, with a significant frustration dose-response. These distress endpoints were secondary and underpowered, so they were reported as suggestive and not as the primary finding. I have since come to realize that despite being a null primary result, this is actually a quite exciting finding. It motivates two ideas, both I now recognize as quite important: Perhaps stability is more critical than valence: the flip fraction results show that, even if the mean for welfare-relevant indicators remains similar, this may be hiding meaningful differences in welfare-relevant circumstances. This would be important because it could be the case that, not only does quantization have a welfare consequence, but also that it could be difficult to notice in the aggregate. Perhaps subdued behavioral change is hiding dissociation: the first study did not measure or probe any internal model representation, but an obvious question is whether the null result conceals a deeper difference, where internal states have changed without a detectable behavior change. This would be important because it could point toward future welfare and alignment issues currently lurking under the surface. Study 2 Study 2 seeks to move in the direction of these ideas, while preserving the original scope from the initial preregistration post; and given the details of that original document, it will clearly not be the last study in the series. It will extend the work to exercise Tier 2, above. The rest of this post will be dedicated to explaining the plan and goals for Study 2 and will serve as a new, dedicated preregistration for its design, endpoints and analysis. Questions There are clearly many things we could imagine asking about representation, behavior and their dissociation; Study 2 focuses on 3 in particular. Does quantization shift the model's representational geometry on welfare-relevant directions even where (per Study 1) t

Read the full article at LessWrong

Related coverage