analiza aiqbench
STRIDE addresses the computational bottleneck of Training Data Attribution by shifting from expensive parameter-space gradient tracking to a more efficient activation-space sparse recovery framework. By utilizing lightweight steering operators, the method achieves state-of-the-art attribution accuracy for Large Language Models while operating 13 times faster than prior approaches. This advancement enables practical scaling of data selection and contamination analysis without the prohibitive costs of repeated retraining.
STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations
arXiv:2606.05165v1 Announce Type: new Abstract: Training Data Attribution (TDA) seeks to trace a model's predictions back to its training data. The gold standard for TDA relies on causal interventions, observing how a model changes when data is added or removed, but repeated retraining is computationally challenging for Large Language Models (LLMs). Consequently, most approaches approximate this effect in the parameter space using gradients. However, tracking gradients across billions of parameters is not only prohibitively expensive but relies on local approximations. In this work, we propose a shift: rather than estimating parameter changes, we model the functional effect of training data in the activation space. We introduce STRIDE (Steering-based Training Data Influence Decomposition), a framework that formulates TDA as a sparse recovery problem in the spirit of compressive sensing. STRIDE learns lightweight "steering operators" that mimic the behavioral shift caused by training on data subsets. By measuring how these operators perturb test predictions, we recover individual training example influences via sparse linear decomposition. STRIDE achieves state-of-the-art for LLM pre-training attribution while being an order of magnitude ($13\times$) faster than previous art. We further validate its practical utility through downstream applications including data selection, data contamination, and qualitative analysis.
Abstract:Training Data Attribution (TDA) seeks to trace a model's predictions back to its training data. The gold standard for TDA relies on causal interventions, observing how a model changes when data is added or removed, but repeated retraining is computationally challenging for Large Language Models (LLMs). Consequently, most approaches approximate this effect in the parameter space using gradients. However, tracking gradients across billions of parameters is not only prohibitively expensive but relies on local approximations. In this work, we propose a shift: rather than estimating parameter changes, we model the functional effect of training data in the activation space. We introduce STRIDE (Steering-based Training Data Influence Decomposition), a framework that formulates TDA as a sparse recovery problem in the spirit of compressive sensing. STRIDE learns lightweight "steering operators" that mimic the behavioral shift caused by training on data subsets. By measuring how these operators perturb test predictions, we recover individual training example influences via sparse linear decomposition. STRIDE achieves state-of-the-art for LLM pre-training attribution while being an order of magnitude ($13\times$) faster than previous art. We further validate its practical utility through downstream applications including data selection, data contamination, and qualitative analysis.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2606.05165 [cs.LG]
(or arXiv:2606.05165v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.05165 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 290 cuvinte
Subiecte conexe
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
PIRAMID: Progress and Plans
LessWrong · 27 iul. 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · 12 iun. 2026
You Only Train Once: Differentiable Subset Selection for Omics Data
arXiv · 5 iun. 2026