aiqbench analysis
This paper offers a layered way to study whether a language model has memorized specific training examples, separating information that can be decoded from internal activations from information the model actually uses in its output. Its most useful finding for practitioners is that membership signals can show up in behavior under reinforcement-learning-based methods without a matching rise in standard likelihood-based inference scores, which suggests current membership-inference audits may miss some forms of training-data exposure. The results come from a specific set of models and methods, so they should be read as a mechanistic lead rather than a settled account of memorization.
From Representation to Readout: Mechanistic Analysis of Membership Signals in Language Models
arXiv:2610.41445v1 Announce Type: new Abstract: Membership inference aims to determine whether a particular example was included in a model's training data. Such membership signals are often broadly described as memorization, while the underlying mechanisms can involve both how information is represented internally and how it is accessed by the model to produce behaviors. We therefore present a mechanistic study of membership signals in large language models (LLMs). We introduce Decodable Information Score, Used Information Score, and Readout Accessibility Gap (DUG), a set of complementary measures that separately quantify membership information decodable from internal representations and the extent to which it is exposed through the model's readout. Across supervised fine-tuning (SFT) and reinforcement learning (RL)-based membership inference methods, membership-specific effects appear in both the amount of decodable information and readout under SFT, while under RL-based membership inference methods they appear primarily in readout. Activation patching further localizes this membership-specific effect to intermediate layers. To examine whether these findings extend to a more general RL setting, we further study reinforcement learning with verifiable rewards (RLVR) contamination, where the training reward is independent of the reference solution. In this setting, we find that behavioral changes can emerge without a corresponding increase in likelihood-based membership inference signals.
Abstract:Membership inference aims to determine whether a particular example was included in a model's training data. Such membership signals are often broadly described as memorization, while the underlying mechanisms can involve both how information is represented internally and how it is accessed by the model to produce behaviors. We therefore present a mechanistic study of membership signals in large language models (LLMs). We introduce Decodable Information Score, Used Information Score, and Readout Accessibility Gap (DUG), a set of complementary measures that separately quantify membership information decodable from internal representations and the extent to which it is exposed through the model's readout. Across supervised fine-tuning (SFT) and reinforcement learning (RL)-based membership inference methods, membership-specific effects appear in both the amount of decodable information and readout under SFT, while under RL-based membership inference methods they appear primarily in readout. Activation patching further localizes this membership-specific effect to intermediate layers. To examine whether these findings extend to a more general RL setting, we further study reinforcement learning with verifiable rewards (RLVR) contamination, where the training reward is independent of the reference solution. In this setting, we find that behavioral changes can emerge without a corresponding increase in likelihood-based membership inference signals.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.41445 [cs.LG]
(or arXiv:2610.41445v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.41445 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 295 words
Related coverage
The Latin Substrate: How Language Models Represent and Mediate Script Choice
arXiv · Sep 9, 2026
Artificial entrepreneurial cognition: Locating and causally steering an opportunity recognition dial inside large language models (LLMs)
arXiv · Sep 15, 2026
StructLens: A Structural Lens for Language Models via Maximum Spanning Trees
arXiv · Oct 22, 2026
Functional Emotion Without Character: Large Language Models, Aristotelian Disposition, and the Limits of Behavioral Alignment
arXiv · Sep 22, 2026