How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum

Title:How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum

Abstract:The standard recipe for training reasoning models---SFT on rationale annotations followed by RLVR---relies on rationale annotations being available and clean. When no annotated rationales exist, SFT cannot start at all, and when the model is too unaligned, RLVR alone stalls at cold start; when annotations are weak, SFT memorizes the errors. Using the Tsallis $q$-logarithm, we define a loss family $J_Q$ that interpolates between RLVR (at $q{=}0$, the exploitation pole) and the log-marginal-likelihood over latent trajectories (at $q{=}1$, the density-estimation pole). The standard pipeline corresponds to a stepwise $q{=}1 \to q{=}0$ schedule, and intermediate $q$ enables training in regimes where the two-step pipeline cannot apply. All members share the same gradient direction, differing only by a per-instance amplification $P_{\theta}^{-q}$ that reweights each instance independently of the learning rate. Under gradient flow analysis, we show that the exploitation pole requires $\Omega(\frac{1}{p_0})$ time to escape cold start but is robust to label noise, while the density-estimation pole escapes in $\Theta\big(\log(\frac{1}{p_0})\big)$ but memorizes label noise. This separation explains why SFT ($q{=}1$) precedes RLVR ($q{=}0$) in the standard pipeline. We further derive two Monte Carlo estimators that directly optimize fixed-$q$ on the $J_Q$ continuum, without annotated rationales or external verifiers: Gradient-Amplified RL (GARL) and Posterior-Attenuated Fine-Tuning (PAFT), with shared bias $O\big(\frac{q}{M P_{\theta}^q}\big)$ but different variance and stability properties. On FinQA, HotPotQA, and MuSiQue, GARL at sufficiently high $q$ escapes cold start where GRPO fails entirely under Qwen 3 and Gemma 4 architectures.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2604.25907 [cs.LG]

(or arXiv:2604.25907v3 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2604.25907 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.