analiza aiqbench
This research addresses a critical gap in data attribution by demonstrating that existing "In-Run" Shapley methods fail when applied to adaptive optimizers like Adam due to their reliance on linear SGD assumptions. The authors introduce an optimizer-aware approach using a Linearized Ghost Approximation, which restores high fidelity to true marginal contributions while maintaining near-standard training throughput. This advancement is vital for practitioners seeking reliable bias mitigation and computational efficiency in modern deep learning pipelines that utilize non-linear optimization dynamics.
In-Run Data Shapley for Adam Optimizer
arXiv:2602.00329v4 Announce Type: replace-cross Abstract: Reliable data attribution is essential for mitigating bias and reducing computational waste in modern machine learning, with the Shapley value serving as the theoretical gold standard. While recent "In-Run" methods bypass the prohibitive cost of retraining by estimating contributions dynamically, they heavily rely on the linear structure of Stochastic Gradient Descent (SGD) and fail to capture the complex dynamics of adaptive optimizers like Adam. In this work, we demonstrate that data attribution is inherently optimizer-dependent: we show that SGD-based proxies diverge significantly from true contributions under Adam (Pearson $R \approx 0.11$), rendering them ineffective for modern training pipelines. To bridge this gap, we propose Adam-Aware In-Run Data Shapley. We derive a closed-form approximation that restores additivity by redefining utility under a fixed-state assumption and enable scalable computation via a novel Linearized Ghost Approximation. This technique linearizes the variance-dependent scaling term, allowing us to compute pairwise gradient dot-products without materializing per-sample gradients. Extensive experiments show that our method achieves near-perfect fidelity to ground-truth marginal contributions ($R > 0.99$) while retaining $\sim$95\% of standard training throughput. Furthermore, our Adam-aware attribution significantly outperforms SGD-based baselines in data attribution downstream tasks.
Abstract:Reliable data attribution is essential for mitigating bias and reducing computational waste in modern machine learning, with the Shapley value serving as the theoretical gold standard. While recent "In-Run" methods bypass the prohibitive cost of retraining by estimating contributions dynamically, they heavily rely on the linear structure of Stochastic Gradient Descent (SGD) and fail to capture the complex dynamics of adaptive optimizers like Adam. In this work, we demonstrate that data attribution is inherently optimizer-dependent: we show that SGD-based proxies diverge significantly from true contributions under Adam (Pearson $R \approx 0.11$), rendering them ineffective for modern training pipelines. To bridge this gap, we propose Adam-Aware In-Run Data Shapley. We derive a closed-form approximation that restores additivity by redefining utility under a fixed-state assumption and enable scalable computation via a novel Linearized Ghost Approximation. This technique linearizes the variance-dependent scaling term, allowing us to compute pairwise gradient dot-products without materializing per-sample gradients. Extensive experiments show that our method achieves near-perfect fidelity to ground-truth marginal contributions ($R > 0.99$) while retaining $\sim$95\% of standard training throughput. Furthermore, our Adam-aware attribution significantly outperforms SGD-based baselines in data attribution downstream tasks.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2602.00329 [cs.LG]
(or arXiv:2602.00329v4 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 201 cuvinte
Subiecte conexe
Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controller
arXiv · 15 oct. 2026
Challenge: Hand coding weights for efficient sequence memorisation
AI Alignment Forum · 23 iul. 2026
Optimizer-dependent training dynamics converge to the same one-third optimal data scaling
arXiv · 30 sept. 2026
Databricks and NVIDIA: Building for the Agentic Era - Cloned
Databricks · 17 iun. 2026