aiqbench analysis
This research introduces a novel method for detecting training data membership by analyzing how models behave under recursive self-generation, distinguishing between anchored training samples and drifting unseen data. The findings demonstrate that this behavioral signal significantly improves privacy risk assessment across various model architectures, offering a more robust alternative to single-pass inference attacks.
Anchored or Drifting: What Recursive Self-Generation Reveals About Training Data
arXiv:2606.31991v2 Announce Type: replace-cross Abstract: Large generative models are known to memorize their training data, posing severe privacy risks. Yet, current methods to detect training membership typically rely on the weak signals of a single forward pass. In this work, we find that training samples and unseen (held-out) data follow visibly different trajectories under recursive self-generation -- repeatedly feeding a model's output back as its next input. Held-out samples \emph{drift}: they lose the specifics of the original within a few steps. Training samples stay \emph{anchored}, degrading far more slowly. We show that the membership signal this produces holds across model modalities, architectures and scales, spanning language, diffusion, and autoregressive vision models. Furthermore, these recursive trajectories provide a signal that raises membership inference TPR at $1\%$ FPR for nearly every attack we evaluate, roughly doubling it on the weakest baselines and still improving the strongest, which shows that a model's behavior under recursion carries membership evidence that a single query does not.
Abstract:Large generative models are known to memorize their training data, posing severe privacy risks. Yet, current methods to detect training membership typically rely on the weak signals of a single forward pass. In this work, we find that training samples and unseen (held-out) data follow visibly different trajectories under recursive self-generation -- repeatedly feeding a model's output back as its next input. Held-out samples \emph{drift}: they lose the specifics of the original within a few steps. Training samples stay \emph{anchored}, degrading far more slowly. We show that the membership signal this produces holds across model modalities, architectures and scales, spanning language, diffusion, and autoregressive vision models. Furthermore, these recursive trajectories provide a signal that raises membership inference TPR at $1\%$ FPR for nearly every attack we evaluate, roughly doubling it on the weakest baselines and still improving the strongest, which shows that a model's behavior under recursion carries membership evidence that a single query does not.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.31991 [cs.LG]
(or arXiv:2606.31991v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.31991 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 251 words
Related coverage
Game Design Creativity evals
LessWrong · Oct 12, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
Top 10 Best Container Registry Security Tools in 2026
Cybersecurity News · Sep 18, 2026
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026