Ascent Conditions for Length-Normalized Single-Rollout Policy Updates

Abstract:Dividing each trajectory's score contribution by its realized length can change the direction of a reward-centered policy update. In a finite trajectory model, we derive an exact ascent threshold for a three-outcome softmax policy and exhibit a binary-reward example in which the expected Euclidean update decreases task success. Natural-gradient ascent holds for fully controllable outcome distributions with deterministic binary labels, but can fail under a one-parameter restriction. A third-order U-statistic estimates the first-order return change without bias from independent current-policy trajectories, one per request. In the example, a same-sample plug-in estimator instead has the opposite sign in expectation for batch sizes two through forty-nine. Exact policy-grid enumeration confirms the local sign reversal. In a branching process, length-normalized learning initially loses return, then recovers and exceeds the unnormalized comparator. The analysis concerns the expected update field at the current policy.

Machine Learning (cs.LG)

arXiv:2610.14670 [cs.LG]

(or arXiv:2610.14670v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.14670 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

reproducibility/README.md

reproducibility/Results/calibration.csv

reproducibility/Results/constant_32_L.npy

reproducibility/Results/constant_32_M.npy

reproducibility/Results/constant_32_P.npy

reproducibility/Results/constant_32_U.npy

reproducibility/Results/constant_64_L.npy

reproducibility/Results/constant_64_M.npy

reproducibility/Results/constant_64_P.npy

reproducibility/Results/constant_64_U.npy

reproducibility/Results/diagnostic.csv

reproducibility/Results/expected_dynamics.csv

reproducibility/Results/experiment_report.json

reproducibility/Results/grid_all.csv

reproducibility/Results/grid_summary.csv

reproducibility/Results/initial_directions.csv

reproducibility/Results/paired_differences.csv

reproducibility/Results/rational_example.json

reproducibility/Results/seed_results.csv

reproducibility/Results/sensitivity_seed_results.csv

reproducibility/Results/sensitivity_summary.csv

reproducibility/Results/summary.csv

reproducibility/Results/variable_32_L.npy

reproducibility/Results/variable_32_M.npy

reproducibility/Results/variable_32_P.npy

reproducibility/Results/variable_32_U.npy

reproducibility/Results/variable_64_L.npy

reproducibility/Results/variable_64_M.npy

reproducibility/Results/variable_64_P.npy

reproducibility/Results/variable_64_U.npy

reproducibility/SHA256.json

reproducibility/Scripts/build_outputs.py

reproducibility/Scripts/diagnostic.py

reproducibility/Scripts/sensitivity.py

reproducibility/Scripts/study.py

reproducibility/Validation/diagnostic_checks.json

reproducibility/Validation/sensitivity_checks.json

reproducibility/Validation/study_checks.json

reproducibility/Validation/theory_checks.json

reproducibility/Validation/theory_checks.py

reproducibility/requirements.txt

(36 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.