aiqbench analysis
This research introduces a blockwise adaptive cubic regularization optimizer designed to overcome the scalability limits of traditional Newton methods in large-scale machine learning. By partitioning parameters and utilizing matrix-free Krylov subspace solvers, the method maintains optimal convergence rates while handling massive tensor models like ViSIR INRs that are typically intractable for full eigendecomposition approaches. The technique demonstrates significant performance gains over standard optimizers such as Adam on complex inverse rendering tasks, offering a robust solution for training high-parameter count models without sacrificing theoretical guarantees.
Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence
arXiv:2608.22129v1 Announce Type: new Abstract: Cubic-regularized Newton methods have the optimal $\mathcal{O}(\varepsilon^{-3/2})$ global rate and an automatic saddle-escape mechanism, but their subproblem is most often solved by a full eigendecomposition, limiting feasible model size. We introduce a blockwise optimizer that partitions parameters by tensor, minimizes an independent cubic model with an adaptive cubic constant $M_b$ per block, and accepts or rejects each block step against a monotone guard on the full loss. The subproblem solver is chosen by block size: small blocks use lazy exact cubic steps from explicitly formed per-block Hessians; arbitrarily large tensors use a matrix-free Chebyshev-bounded Krylov subspace built by the Lanczos process. The cubic shift bounds the required polynomial degree whenever the gradient-driven shift dominates negative curvature, renders the shifted operator positive semidefinite before any polynomial is applied, and preserves the $\mathcal{O}(\varepsilon^{-3/2})$ rate under inexact subproblem solves. We prove these claims, and the blockwise scheme carries a monotone per-block descent guarantee. Experiments cover FINER INRs (about 199k parameters) and a 91.4M-parameter ViSIR INR, where the blockwise cubic step remains exact in the cubic-model sense on every block, including the 88.5M-parameter decoder tensor (97\% of the model). Run to full convergence on FINER, the ARC-$\varphi_1$ optimizer reaches 133.5 dB PSNR while tuned Adam plateaus at 78.2 dB at the same extended budget; in the roughly 70 minutes Adam takes to reach its peak, ARC-$\varphi_1$ reaches 95.6 dB. A companion report isolates the loss-landscape features responsible for Adam's behavior.
Abstract:Cubic-regularized Newton methods have the optimal $\mathcal{O}(\varepsilon^{-3/2})$ global rate and an automatic saddle-escape mechanism, but their subproblem is most often solved by a full eigendecomposition, limiting feasible model size. We introduce a blockwise optimizer that partitions parameters by tensor, minimizes an independent cubic model with an adaptive cubic constant $M_b$ per block, and accepts or rejects each block step against a monotone guard on the full loss. The subproblem solver is chosen by block size: small blocks use lazy exact cubic steps from explicitly formed per-block Hessians; arbitrarily large tensors use a matrix-free Chebyshev-bounded Krylov subspace built by the Lanczos process. The cubic shift bounds the required polynomial degree whenever the gradient-driven shift dominates negative curvature, renders the shifted operator positive semidefinite before any polynomial is applied, and preserves the $\mathcal{O}(\varepsilon^{-3/2})$ rate under inexact subproblem solves. We prove these claims, and the blockwise scheme carries a monotone per-block descent guarantee. Experiments cover FINER INRs (about 199k parameters) and a 91.4M-parameter ViSIR INR, where the blockwise cubic step remains exact in the cubic-model sense on every block, including the 88.5M-parameter decoder tensor (97\% of the model). Run to full convergence on FINER, the ARC-$\varphi_1$ optimizer reaches 133.5 dB PSNR while tuned Adam plateaus at 78.2 dB at the same extended budget; in the roughly 70 minutes Adam takes to reach its peak, ARC-$\varphi_1$ reaches 95.6 dB. A companion report isolates the loss-landscape features responsible for Adam's behavior.
Machine Learning (cs.LG); Numerical Analysis (math.NA)
arXiv:2608.22129 [cs.LG]
(or arXiv:2608.22129v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.22129 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 333 words
Related coverage
Physics-Guided Spatiotemporal Learning for Coastal Wave Peak Period Estimation from Video
arXiv · Jun 12, 2026
Freeze, Then Select: Structured Field Adapters and Stability-Validated Weak Selection for PDE Discovery from Sparse Observations
arXiv · Aug 3, 2026
Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory
arXiv · Jul 24, 2026
Adaptive Rollout Truncation Based on Epistemic Uncertainty for Efficient Offline World Model Training
arXiv · Sep 21, 2026