analiza aiqbench
BLOOM-WILT introduces a novel logit tilting technique that significantly improves the sample efficiency of automated LLM auditors by adaptively reweighting token probabilities to surface rare safety behaviors. This method effectively bridges the gap between limited evaluation scales and real-world deployment interactions, overturning previous safety rankings without requiring additional training costs. For researchers in AI safety and alignment, this approach offers a scalable pathway to rigorously test deployed models against edge-case scenarios that standard testing fails to capture.
BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
arXiv:2608.31105v1 Announce Type: new Abstract: Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate. Automated auditors make testing cheap to scale and flexible enough to cover almost any specified behaviour, yet their lack of optimisation pressure makes them sample-inefficient. To address this shortcoming, we introduce BLOOM-WILT, a full auditing pipeline that elicits natural multi-turn instances of rare behaviours, without training cost or access beyond the target's next-token distribution. On the input side, WILT's auditor model revises its conversational strategy across rounds, learning from previous scored interactions. On the output side, WILT adaptively reweights the target's decoding using the model's own distribution conditioned on an elicitation prompt, so that behaviour-relevant generations are sampled ahead of others it finds equally probable when unprompted. We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings and overturns the previous model safety rankings. WILT raises average behaviour presence from 51% to 100% when eliciting self-harm encouragement from Qwen3.5-4B, beating every elicitation method we port into the same pipeline at matched compute, without pushing output probability below the baseline's.
Abstract:Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate. Automated auditors make testing cheap to scale and flexible enough to cover almost any specified behaviour, yet their lack of optimisation pressure makes them sample-inefficient. To address this shortcoming, we introduce BLOOM-WILT, a full auditing pipeline that elicits natural multi-turn instances of rare behaviours, without training cost or access beyond the target's next-token distribution. On the input side, WILT's auditor model revises its conversational strategy across rounds, learning from previous scored interactions. On the output side, WILT adaptively reweights the target's decoding using the model's own distribution conditioned on an elicitation prompt, so that behaviour-relevant generations are sampled ahead of others it finds equally probable when unprompted. We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings and overturns the previous model safety rankings. WILT raises average behaviour presence from 51% to 100% when eliciting self-harm encouragement from Qwen3.5-4B, beating every elicitation method we port into the same pipeline at matched compute, without pushing output probability below the baseline's.
10 pages, 5 figures, 3 tables. Code: this https URL
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2608.31105 [cs.AI]
(or arXiv:2608.31105v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.31105 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 309 cuvinte
Subiecte conexe
State of the Field: AI for Epistemics and Coordination
LessWrong · 2 oct. 2026
The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology
LessWrong · 23 iul. 2026
Prism: Automating Science-of-Evals Research
AI Alignment Forum · 13 iul. 2026
Audit tools of the trade: Automated Auditor Elicitation Can Be Sensitive to the Context the Auditor Is Given
arXiv · 15 oct. 2026