analiza aiqbench
This paper is a theoretical machine learning study of policy gradient methods, showing that dividing each trajectory's score by its length can reverse the sign of the expected update and reduce task success in some settings. It also proposes a third-order U-statistic estimator that is unbiased for the first-order return change, while a common same-sample plug-in estimator can have the wrong sign for small batches. Practitioners who length-normalize rewards or log-probabilities in RL fine-tuning should note that the result is a local, expected-update analysis on constructed examples, not a general claim about training performance.
Ascent Conditions for Length-Normalized Single-Rollout Policy Updates
arXiv:2610.14670v1 Announce Type: new Abstract: Dividing each trajectory's score contribution by its realized length can change the direction of a reward-centered policy update. In a finite trajectory model, we derive an exact ascent threshold for a three-outcome softmax policy and exhibit a binary-reward example in which the expected Euclidean update decreases task success. Natural-gradient ascent holds for fully controllable outcome distributions with deterministic binary labels, but can fail under a one-parameter restriction. A third-order U-statistic estimates the first-order return change without bias from independent current-policy trajectories, one per request. In the example, a same-sample plug-in estimator instead has the opposite sign in expectation for batch sizes two through forty-nine. Exact policy-grid enumeration confirms the local sign reversal. In a branching process, length-normalized learning initially loses return, then recovers and exceeds the unnormalized comparator. The analysis concerns the expected update field at the current policy.
Abstract:Dividing each trajectory's score contribution by its realized length can change the direction of a reward-centered policy update. In a finite trajectory model, we derive an exact ascent threshold for a three-outcome softmax policy and exhibit a binary-reward example in which the expected Euclidean update decreases task success. Natural-gradient ascent holds for fully controllable outcome distributions with deterministic binary labels, but can fail under a one-parameter restriction. A third-order U-statistic estimates the first-order return change without bias from independent current-policy trajectories, one per request. In the example, a same-sample plug-in estimator instead has the opposite sign in expectation for batch sizes two through forty-nine. Exact policy-grid enumeration confirms the local sign reversal. In a branching process, length-normalized learning initially loses return, then recovers and exceeds the unnormalized comparator. The analysis concerns the expected update field at the current policy.
Machine Learning (cs.LG)
arXiv:2610.14670 [cs.LG]
(or arXiv:2610.14670v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.14670 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
reproducibility/README.md
reproducibility/Results/calibration.csv
reproducibility/Results/constant_32_L.npy
reproducibility/Results/constant_32_M.npy
reproducibility/Results/constant_32_P.npy
reproducibility/Results/constant_32_U.npy
reproducibility/Results/constant_64_L.npy
reproducibility/Results/constant_64_M.npy
reproducibility/Results/constant_64_P.npy
reproducibility/Results/constant_64_U.npy
reproducibility/Results/diagnostic.csv
reproducibility/Results/expected_dynamics.csv
reproducibility/Results/experiment_report.json
reproducibility/Results/grid_all.csv
reproducibility/Results/grid_summary.csv
reproducibility/Results/initial_directions.csv
reproducibility/Results/paired_differences.csv
reproducibility/Results/rational_example.json
reproducibility/Results/seed_results.csv
reproducibility/Results/sensitivity_seed_results.csv
reproducibility/Results/sensitivity_summary.csv
reproducibility/Results/summary.csv
reproducibility/Results/variable_32_L.npy
reproducibility/Results/variable_32_M.npy
reproducibility/Results/variable_32_P.npy
reproducibility/Results/variable_32_U.npy
reproducibility/Results/variable_64_L.npy
reproducibility/Results/variable_64_M.npy
reproducibility/Results/variable_64_P.npy
reproducibility/Results/variable_64_U.npy
reproducibility/SHA256.json
reproducibility/Scripts/build_outputs.py
reproducibility/Scripts/diagnostic.py
reproducibility/Scripts/sensitivity.py
reproducibility/Scripts/study.py
reproducibility/Validation/diagnostic_checks.json
reproducibility/Validation/sensitivity_checks.json
reproducibility/Validation/study_checks.json
reproducibility/Validation/theory_checks.json
reproducibility/Validation/theory_checks.py
reproducibility/requirements.txt
(36 additional files not shown) You must enabled JavaScript to view entire file list.
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 288 cuvinte
Subiecte conexe
When Partial Credit Opposes Completion
Snorkel AI Blog · 15 oct. 2026
Debugging Logprob Mismatches in LLM Reinforcement Learning
AMD · 23 sept. 2026
[AINews] not much happened today
Latent Space · 10 sept. 2026
Scaling RL with verl on AMD Instinct MI355X: Async Walkthrough and Sync Benchmark
AMD · 18 aug. 2026