Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamics

Abstract:Prior work has shown that neural networks exhibit implicit biases toward low-complexity structure (e.g., spectral bias), memorization dynamics, and compression-like effects during training, but a unified dynamical account of selective retention remains incomplete. We propose Repeated Reinforcement with Persistent Forgetting (RPF) dynamics, a minimal framework in which repeated exposure reinforces patterns and structures that recur in the data, while persistent forgetting attenuates learned information. This view treats forgetting not merely as a failure mode, but as a selection mechanism. We build the theory in three successive layers. First, in an independent-feature model, we derive an exposure-selective survival law and a support-dependent retention boundary characterizing which patterns persist under forgetting. Second, in a shared-parameter model, we show that forgetting induces spectral filtering over covariance modes, preserving strongly supported shared components while suppressing weak ones. Third, under small-step and norm/coding approximations, we show how RPF dynamics induce an implicit trade-off between data fitting and the cost of stored information, yielding Minimum Description Length (MDL)-like compression. Controlled experiments provide evidence for this reinforcement--forgetting selection mechanism in scalar memories and a nonlinear shared network. Joint reinforcement and attenuation interventions shift conditional retention, while matched exposure counts reveal forgetting-dependent effects of reinforcement timing and changes in the composition of the retained set. Together, these results show that repeated reinforcement and persistent forgetting jointly provide a controllable source of inductive bias beyond neural architecture and scale.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2609.38768 [cs.LG]

(or arXiv:2609.38768v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2609.38768 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

experimental_evidence/README.txt

experimental_evidence/SHA256SUMS.json

experimental_evidence/conditional_tinylm/conditional_phase_diagram.py

experimental_evidence/conditional_tinylm/coupling_analysis.py

experimental_evidence/conditional_tinylm/figures/coupling/competition_models.png

experimental_evidence/conditional_tinylm/figures/coupling/factorial_interaction.png

experimental_evidence/conditional_tinylm/figures/full/boundary_scaling.png

experimental_evidence/conditional_tinylm/figures/full/conditional_control.png

experimental_evidence/conditional_tinylm/figures/full/phase_diagram.png

experimental_evidence/conditional_tinylm/figures/full/ratio_organization.png

experimental_evidence/conditional_tinylm/figures/full/support_curves.png

experimental_evidence/conditional_tinylm/figures/schedule_coupling/selection_tradeoff.png

experimental_evidence/conditional_tinylm/figures/schedule_coupling/temporal_coupling.png

experimental_evidence/conditional_tinylm/results/coupling/coupling_summary.json

experimental_evidence/conditional_tinylm/results/coupling/cross_validation_folds.csv

experimental_evidence/conditional_tinylm/results/coupling/cross_validation_summary.csv

experimental_evidence/conditional_tinylm/results/coupling/interaction_cells.csv

experimental_evidence/conditional_tinylm/results/coupling/interaction_model_sensitivity.csv

experimental_evidence/conditional_tinylm/results/coupling/replay_gain_by_decay.csv

experimental_evidence/conditional_tinylm/results/full/analysis_summary.json

experimental_evidence/conditional_tinylm/results/full/config.json

experimental_evidence/conditional_tinylm/results/full/environment.json

experimental_evidence/conditional_tinylm/results/full/phase_table.csv

experimental_evidence/conditional_tinylm/results/full/run_summary.csv

experimental_evidence/conditional_tinylm/results/full/threshold_sensitivity.csv

experimental_evidence/conditional_tinylm/results/full/token_metrics.csv

experimental_evidence/conditional_tinylm/results/schedule_coupling/analysis_summary.json

experimental_evidence/conditional_tinylm/results/schedule_coupling/config.json

experimental_evidence/conditional_tinylm/results/schedule_coupling/environment.json

experimental_evidence/conditional_tinylm/results/schedule_coupling/paired_contrast_summary.csv

experimental_evidence/conditional_tinylm/results/schedule_coupling/paired_contrasts.csv

experimental_evidence/conditional_tinylm/results/schedule_coupling/run_summary.csv

experimental_evidence/conditional_tinylm/results/schedule_coupling/sample_counts.csv

experimental_evidence/conditional_tinylm/results/schedule_coupling/sequence_metadata.json

experimental_evidence/conditional_tinylm/results/schedule_coupling/token_metrics.csv

experimental_evidence/conditional_tinylm/results/schedule_coupling/trajectory.csv

experimental_evidence/conditional_tinylm/schedule_coupling_experiment.py

experimental_evidence/conditional_tinylm/verify_coupling_results.py

experimental_evidence/conditional_tinylm/verify_results.py

experimental_evidence/conditional_tinylm/verify_theory_coupling.py

experimental_evidence/factorized/code/factorized_generalization_experiment.py

experimental_evidence/factorized/figures/factorized_generalization.png

experimental_evidence/factorized/results/factorized/analysis_summary.json

experimental_evidence/factorized/results/factorized/config.json

experimental_evidence/factorized/results/factorized/environment.json

experimental_evidence/factorized/results/factorized/pair_design.csv

experimental_evidence/factorized/results/factorized/paired_replay_summary.csv

experimental_evidence/factorized/results/factorized/run_summary.csv

experimental_evidence/reproduce_figures/scripts/build_figures_tables.py

(44 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.