Gaussian Limits for SGD Without Stationary Moments

Abstract:Temporal dependence can separate the Gaussian approximation of stochastic gradient descent from its stationary moments. For unmodified least-squares SGD, we construct a design with standard Gaussian marginals whose stationary error has every positive moment infinite. Independent observations with the same marginals instead give finite stationary variance. Both regimes retain a Gaussian small-step limit. Our general theory establishes pathwise contraction from a finite second design moment, then uses score cancellation and localization to obtain stationary Gaussian and Ornstein--Uhlenbeck limits. Independent Gaussian regression errors yield an exact conditional Gaussian law and total-variation convergence under the same design integrability. Stronger design conditions identify a positive first-order total-variation constant and a deterministic covariance correction with $o(a)$ error. A scalar coverage expansion translates this correction into its inference consequence. Experiments examine distributional error, coverage, and calibration with dependent scores. Together, these results establish precise probability-law approximation beyond moment-based stationary analysis.

Machine Learning (stat.ML); Machine Learning (cs.LG)

arXiv:2610.05599 [stat.ML]

(or arXiv:2610.05599v1 [stat.ML] for this version)

https://doi.org/10.48550/arXiv.2610.05599 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

experiments/__init__.py

experiments/_panel_alignment.py

experiments/auxiliary.py

experiments/configs/extensions.json

experiments/configs/full.json

experiments/configs/generate.py

experiments/diagnostics.py

experiments/direct_evidence.py

experiments/extensions.py

experiments/make_figures.py

experiments/models.py

experiments/non_gaussian.py

experiments/population_bartlett.py

experiments/run_experiments.py

experiments/scalar_gaussian.py

experiments/summarize.py

experiments/test_direct_evidence.py

experiments/test_numerics.py

experiments/tv_sampling_diagnostic.py

experiments/validate_direct_evidence.py

experiments/vector_sgd.py

requirements-lock.txt

requirements.txt

results/all_summaries.csv

results/auxiliary/lower_log10_amplification.npy

results/auxiliary/observed_log10_amplification.npy

results/auxiliary/shocks.npy

results/auxiliary/startup.json

results/direct_evidence/acceptance.json

results/direct_evidence/refresh.json

results/direct_evidence/refresh/r0.7_a0.0025.json

results/direct_evidence/refresh/r0.7_a0.005.json

results/direct_evidence/refresh/r0.7_a0.01.json

results/direct_evidence/refresh/r0.7_a0.02.json

results/direct_evidence/refresh/r0.7_a0.04.json

results/direct_evidence/refresh/r0.9_a0.0025.json

results/direct_evidence/refresh/r0.9_a0.005.json

results/direct_evidence/refresh/r0.9_a0.01.json

results/direct_evidence/refresh/r0.9_a0.02.json

results/direct_evidence/refresh/r0.9_a0.04.json

results/direct_evidence/refresh/r0_a0.0025.json

results/direct_evidence/refresh/r0_a0.005.json

results/direct_evidence/refresh/r0_a0.01.json

results/direct_evidence/refresh/r0_a0.02.json

results/direct_evidence/refresh/r0_a0.04.json

results/direct_evidence/sampling_diagnostic.json

results/direct_evidence/startup.json

results/errors_gaussian_a0.0025/metadata.json

results/errors_gaussian_a0.0025/summary.json

results/errors_gaussian_a0.01/metadata.json

results/errors_gaussian_a0.01/summary.json

results/errors_gaussian_a0.04/metadata.json

results/errors_gaussian_a0.04/summary.json

results/errors_rademacher_a0.0025/metadata.json

results/errors_rademacher_a0.0025/summary.json

results/errors_rademacher_a0.01/metadata.json

results/errors_rademacher_a0.01/summary.json

results/errors_rademacher_a0.04/metadata.json

results/errors_rademacher_a0.04/summary.json

results/errors_student5_a0.0025/metadata.json

results/errors_student5_a0.0025/summary.json

results/errors_student5_a0.01/metadata.json

results/errors_student5_a0.01/summary.json

results/errors_student5_a0.04/metadata.json

results/errors_student5_a0.04/summary.json

results/hac_a0.0025/metadata.json

results/hac_a0.0025/summary.json

results/hac_a0.01/metadata.json

results/hac_a0.01/summary.json

results/hac_a0.04/metadata.json

results/hac_a0.04/summary.json

results/logweibull_c0.5_a0.0025/metadata.json

results/logweibull_c0.5_a0.0025/summary.json

results/logweibull_c0.5_a0.01/metadata.json

results/logweibull_c0.5_a0.01/summary.json

results/logweibull_c0.5_a0.04/metadata.json

results/logweibull_c0.5_a0.04/summary.json

results/logweibull_c1_a0.0025/metadata.json

results/logweibull_c1_a0.0025/summary.json

results/logweibull_c1_a0.01/metadata.json

results/logweibull_c1_a0.01/summary.json

results/logweibull_c1_a0.04/metadata.json

results/logweibull_c1_a0.04/summary.json

results/pareto_r0.7_a0.0025_s0/metadata.json

results/pareto_r0.7_a0.0025_s0/summary.json

results/pareto_r0.7_a0.005_s0/metadata.json

results/pareto_r0.7_a0.005_s0/summary.json

results/pareto_r0.7_a0.01_s0/metadata.json

results/pareto_r0.7_a0.01_s0/summary.json

results/pareto_r0.7_a0.02_s0/metadata.json

results/pareto_r0.7_a0.02_s0/summary.json

results/pareto_r0.7_a0.04_s0/metadata.json

results/pareto_r0.7_a0.04_s0/summary.json

results/pareto_r0.9_a0.0025_s0/metadata.json

results/pareto_r0.9_a0.0025_s0/summary.json

results/pareto_r0.9_a0.005_s0/metadata.json

results/pareto_r0.9_a0.005_s0/summary.json

results/pareto_r0.9_a0.01_s0/metadata.json

results/pareto_r0.9_a0.01_s0/summary.json

results/pareto_r0.9_a0.02_s0/metadata.json

results/pareto_r0.9_a0.02_s0/summary.json

results/pareto_r0.9_a0.04_s0/metadata.json

results/pareto_r0.9_a0.04_s0/summary.json

results/pareto_r0_a0.0025_s0/metadata.json

results/pareto_r0_a0.0025_s0/summary.json

results/pareto_r0_a0.005_s0/metadata.json

results/pareto_r0_a0.005_s0/summary.json

results/pareto_r0_a0.01_s0/metadata.json

results/pareto_r0_a0.01_s0/summary.json

results/pareto_r0_a0.02_s0/metadata.json

results/pareto_r0_a0.02_s0/summary.json

results/pareto_r0_a0.04_s0/metadata.json

results/pareto_r0_a0.04_s0/summary.json

results/pooled/refresh_r0.7_a0.0025/summary.json

results/pooled/refresh_r0.7_a0.005/summary.json

results/pooled/refresh_r0.7_a0.01/summary.json

results/pooled/refresh_r0.7_a0.02/summary.json

results/pooled/refresh_r0.7_a0.04/summary.json

results/pooled/refresh_r0.9_a0.0025/summary.json

results/pooled/refresh_r0.9_a0.005/summary.json

results/pooled/refresh_r0.9_a0.01/summary.json

results/pooled/refresh_r0.9_a0.02/summary.json

results/pooled/refresh_r0.9_a0.04/summary.json

results/pooled/refresh_r0_a0.0025/summary.json

results/pooled/refresh_r0_a0.005/summary.json

results/pooled/refresh_r0_a0.01/summary.json

results/pooled/refresh_r0_a0.02/summary.json

results/pooled/refresh_r0_a0.04/summary.json

results/population_bartlett/coverage.csv

results/population_bartlett/summary.json

results/refresh_r0.7_a0.0025_s0/metadata.json

results/refresh_r0.7_a0.0025_s0/summary.json

results/refresh_r0.7_a0.0025_s1/metadata.json

results/refresh_r0.7_a0.0025_s1/summary.json

results/refresh_r0.7_a0.005_s0/metadata.json

results/refresh_r0.7_a0.005_s0/summary.json

results/refresh_r0.7_a0.005_s1/metadata.json

results/refresh_r0.7_a0.005_s1/summary.json

results/refresh_r0.7_a0.01_s0/metadata.json

results/refresh_r0.7_a0.01_s0/summary.json

results/refresh_r0.7_a0.01_s1/metadata.json

results/refresh_r0.7_a0.01_s1/summary.json

results/refresh_r0.7_a0.02_s0/metadata.json

results/refresh_r0.7_a0.02_s0/summary.json

results/refresh_r0.7_a0.02_s1/metadata.json

results/refresh_r0.7_a0.02_s1/summary.json

results/refresh_r0.7_a0.04_s0/metadata.json

results/refresh_r0.7_a0.04_s0/summary.json

results/refresh_r0.7_a0.04_s1/metadata.json

results/refresh_r0.7_a0.04_s1/summary.json

results/refresh_r0.9_a0.0025_s0/metadata.json

results/refresh_r0.9_a0.0025_s0/summary.json

results/refresh_r0.9_a0.0025_s1/metadata.json

results/refresh_r0.9_a0.0025_s1/summary.json

results/refresh_r0.9_a0.005_s0/metadata.json

results/refresh_r0.9_a0.005_s0/summary.json

results/refresh_r0.9_a0.005_s1/metadata.json

results/refresh_r0.9_a0.005_s1/summary.json

results/refresh_r0.9_a0.01_s0/metadata.json

results/refresh_r0.9_a0.01_s0/summary.json

results/refresh_r0.9_a0.01_s1/metadata.json

results/refresh_r0.9_a0.01_s1/summary.json

results/refresh_r0.9_a0.02_s0/metadata.json

results/refresh_r0.9_a0.02_s0/summary.json

results/refresh_r0.9_a0.02_s1/metadata.json

results/refresh_r0.9_a0.02_s1/summary.json

results/refresh_r0.9_a0.04_s0/metadata.json

results/refresh_r0.9_a0.04_s0/summary.json

results/refresh_r0.9_a0.04_s1/metadata.json

results/refresh_r0.9_a0.04_s1/summary.json

results/refresh_r0_a0.0025_s0/metadata.json

results/refresh_r0_a0.0025_s0/summary.json

results/refresh_r0_a0.0025_s1/metadata.json

results/refresh_r0_a0.0025_s1/summary.json

results/refresh_r0_a0.005_s0/metadata.json

results/refresh_r0_a0.005_s0/summary.json

results/refresh_r0_a0.005_s1/metadata.json

results/refresh_r0_a0.005_s1/summary.json

results/refresh_r0_a0.01_s0/metadata.json

results/refresh_r0_a0.01_s0/summary.json

results/refresh_r0_a0.01_s1/metadata.json

results/refresh_r0_a0.01_s1/summary.json

results/refresh_r0_a0.02_s0/metadata.json

results/refresh_r0_a0.02_s0/summary.json

results/refresh_r0_a0.02_s1/metadata.json

results/refresh_r0_a0.02_s1/summary.json

results/refresh_r0_a0.04_s0/metadata.json

results/refresh_r0_a0.04_s0/summary.json

results/refresh_r0_a0.04_s1/metadata.json

results/refresh_r0_a0.04_s1/summary.json

results/stress_logweibull_a0.000625/metadata.json

results/stress_logweibull_a0.000625/summary.json

results/stress_logweibull_a0.0025/metadata.json

results/stress_logweibull_a0.0025/summary.json

results/vector_a0.0025/metadata.json

results/vector_a0.0025/summary.json

results/vector_a0.005/metadata.json

results/vector_a0.005/summary.json

results/vector_a0.01/metadata.json

results/vector_a0.01/summary.json

results/vector_a0.02/metadata.json

results/vector_a0.02/summary.json

results/vector_a0.04/metadata.json

results/vector_a0.04/summary.json

(200 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.