analiza aiqbench
This paper provides a theoretical clarification that Optuna's constrained Tree-Structured Parzen Estimator (c-TPE) effectively implements a joint density formulation rather than the independence assumption found in prior c-TPE literature. The analysis highlights a key practical advantage where this joint approach remains invariant to constraint duplication, preventing the performance degradation seen in independent formulations. For practitioners, this offers a deeper understanding of the algorithmic behavior of widely used hyperparameter optimization tools under complex constraint scenarios.
Optuna Constrained Tree-Structured Parzen Estimator Is a Joint Density Generalization of c-TPE
arXiv:2606.09889v1 Announce Type: new Abstract: Constrained hyperparameter optimization (HPO) is common in practice, yet Optuna's widely used constrained TPE lacks algorithmic analysis. While c-TPE proposes an expected constrained improvement (ECI) approach assuming independence between the objective and constraints, Optuna uses a single joint density over both. We show that Optuna's constrained TPE is joint c-TPE -- the same ECI acquisition function using a joint likelihood. We demonstrate joint c-TPE is invariant to constraint duplication whereas independent c-TPE degrades as the product accumulates duplicated factors. We outline practical tradeoffs between the formulations and directions for future study.
Abstract:Constrained hyperparameter optimization (HPO) is common in practice, yet Optuna's widely used constrained TPE lacks algorithmic analysis. While c-TPE proposes an expected constrained improvement (ECI) approach assuming independence between the objective and constraints, Optuna uses a single joint density over both. We show that Optuna's constrained TPE is joint c-TPE -- the same ECI acquisition function using a joint likelihood. We demonstrate joint c-TPE is invariant to constraint duplication whereas independent c-TPE degrades as the product accumulates duplicated factors. We outline practical tradeoffs between the formulations and directions for future study.
Machine Learning (cs.LG)
arXiv:2606.09889 [cs.LG]
(or arXiv:2606.09889v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.09889 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 183 cuvinte