analiza aiqbench
This paper proposes a numerical optimization approach using damped Gauss-Newton methods to address multi-objective hyperparameter tuning, specifically targeting the underdetermined regime where parameters outnumber objectives. While the method achieves competitive performance on standard datasets by estimating local sensitivity via finite-difference Jacobians, it does not consistently outperform established baselines like random search or TPE across all tested scenarios. The work offers a rigorous mathematical alternative to black-box optimizers but highlights significant costs associated with Jacobian estimation and sensitivity to initialization.
Multi-Objective Hyperparameter Search via Damped Gauss--Newton Optimization
arXiv:2401.03580v2 Announce Type: replace-cross Abstract: We study hyperparameter optimization (HPO) from a numerical-optimization perspective and propose a multi-objective, damped Gauss--Newton search method. Rather than treating model evaluations as independent trials, the method estimates a finite-difference Jacobian that captures the local sensitivity of multiple validation metrics to hyperparameter perturbations. A Tikhonov-regularized Gauss--Newton system then produces a directed joint update, addressing the underdetermined setting in which the number of hyperparameters exceeds the number of performance objectives. We evaluate the method on three public classification datasets by tuning four XGBoost hyperparameters and compare it with exhaustive grid search, random search, and tree-structured Parzen estimator (TPE) optimization. On a controlled Breast Cancer split, the proposed method matches the best validation accuracy of a 320-configuration grid search while obtaining slightly better log loss and ROC--AUC. Across three datasets and three seeds, performance is competitive but does not consistently exceed random search or TPE. Ablations further show sensitivity to damping and initialization. These results position damped Gauss--Newton search as a directed local alternative to exhaustive HPO, while also clarifying the additional model-evaluation cost required for finite-difference Jacobian estimation.
Abstract:We study hyperparameter optimization (HPO) from a numerical-optimization perspective and propose a multi-objective, damped Gauss--Newton search method. Rather than treating model evaluations as independent trials, the method estimates a finite-difference Jacobian that captures the local sensitivity of multiple validation metrics to hyperparameter perturbations. A Tikhonov-regularized Gauss--Newton system then produces a directed joint update, addressing the underdetermined setting in which the number of hyperparameters exceeds the number of performance objectives. We evaluate the method on three public classification datasets by tuning four XGBoost hyperparameters and compare it with exhaustive grid search, random search, and tree-structured Parzen estimator (TPE) optimization. On a controlled Breast Cancer split, the proposed method matches the best validation accuracy of a 320-configuration grid search while obtaining slightly better log loss and ROC--AUC. Across three datasets and three seeds, performance is competitive but does not consistently exceed random search or TPE. Ablations further show sensitivity to damping and initialization. These results position damped Gauss--Newton search as a directed local alternative to exhaustive HPO, while also clarifying the additional model-evaluation cost required for finite-difference Jacobian estimation.
Optimization and Control (math.OC); Machine Learning (cs.LG)
arXiv:2401.03580 [math.OC]
(or arXiv:2401.03580v2 [math.OC] for this version)
https://doi.org/10.48550/arXiv.2401.03580 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 273 cuvinte
Subiecte conexe
Faster Scientific Image Analysis with NVIDIA cuPhoton
NVIDIA · 7 oct. 2026
Calibrating Prediction Timeliness Through Multi-Objective Hyperparameter Optimization for Remaining Useful Life Prediction
arXiv · 2 oct. 2026
Granite 4.2 LLMs: How They're Built
Hugging Face · 25 aug. 2026
Adaptive Experimentation with Meta’s Ax: A Practical Coding Guide
MarkTechPost · 6 aug. 2026