aiqbench analysis
This research evaluates Large Language Model steering techniques by measuring their alignment with human cognition using triadic similarity judgments, rather than relying solely on task-specific metrics. The study finds that prompt-based methods offer superior accuracy and better model-to-human alignment, while also revealing inherent biases in LLMs toward judging conceptual "kind" over physical "size." This work is significant for understanding the representational structures of steered models and improving their interpretability relative to human mental representations.
Evaluating Steering Techniques using Human Similarity Judgments
From arXiv
Full textarXiv:2505.19333v2 Announce Type: replace Abstract: Current evaluations of Large Language Model (LLM) steering techniques focus on task-specific performance, overlooking how well steered representations align with human cognition. Using a well-established triadic similarity judgment task, we assessed steered LLMs on their ability to flexibly judge similarity between concepts based on size or kind, two central dimensions organizing human mental representations. We found that prompt-based steering methods outperformed other methods both in terms of steering accuracy and model-to-human alignment.
We also found LLMs were biased towards `kind' similarity and struggled with `size' alignment. This evaluation approach, grounded in human cognition, adds further support to the efficacy of prompt-based steering and reveals privileged representational axes in LLMs prior to steering.
Abstract:Current evaluations of Large Language Model (LLM) steering techniques focus on task-specific performance, overlooking how well steered representations align with human cognition. Using a well-established triadic similarity judgment task, we assessed steered LLMs on their ability to flexibly judge similarity between concepts based on size or kind, two central dimensions organizing human mental representations. We found that prompt-based steering methods outperformed other methods both in terms of steering accuracy and model-to-human alignment.
We also found LLMs were biased towards `kind' similarity and struggled with `size' alignment. This evaluation approach, grounded in human cognition, adds further support to the efficacy of prompt-based steering and reveals privileged representational axes in LLMs prior to steering.
Artificial Intelligence (cs.AI)
arXiv:2505.19333 [cs.AI]
(or arXiv:2505.19333v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2505.19333 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 204 words
Related coverage
Evaluating the accuracy of KV cache reuse techniques
arXiv · Sep 28, 2026
Span-Level Machine Translation Meta-Evaluation
arXiv · Oct 21, 2026
Does preservation make sense before we know how to revive?
LessWrong · Jun 15, 2026
Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffolds
arXiv · Sep 3, 2026