aiqbench analysis
This arXiv preprint proposes PriorTune, which uses synthetic datasets from a published PFN prior to select hyperparameter configurations for conventional tabular models like LightGBM, rather than for pretraining foundation models. The claimed benefit is that selection happens offline, so new datasets incur no extra tuning cost. Readers should note that the headline result (ranking above TabPFN-3 and RealMLP on BeyondArena) comes from a single benchmark, so its generality and the strength of the reported scaling-law explanation still need independent replication.
Beyond Foundation Models: PFN Priors Improve Tabular Hyperparameter Optimization
arXiv:2611.05141v1 Announce Type: new Abstract: Synthetic priors have so far served one purpose: pretraining tabular foundation models. We ask whether the same priors can improve conventional tabular models, which are trained from scratch on each dataset. We propose PriorTune, a method that meta-tunes hyperparameter configurations on synthetic tasks alone. Offline, for a given model, we fit a pool of configurations on datasets sampled from a published PFN prior and greedily select the subset with the highest winrate over the random baseline. Online, a new dataset fits only the selected configurations, so the gain comes at no extra cost. Empirically, we observe that scaling both the number of configurations and synthetic datasets has an equal scaling effect on the performance of the final subset of configurations, which we also explain theoretically. On the full BeyondArena benchmark, LightGBM with TabPrep features with our configurations ranks above TabPFN-3 and RealMLP, while being much more efficient at inference. Overall, our results establish existing PFN priors as a scalable source of meta-training data for hyperparameter selection for non-foundational tabular models.
Abstract:Synthetic priors have so far served one purpose: pretraining tabular foundation models. We ask whether the same priors can improve conventional tabular models, which are trained from scratch on each dataset. We propose PriorTune, a method that meta-tunes hyperparameter configurations on synthetic tasks alone. Offline, for a given model, we fit a pool of configurations on datasets sampled from a published PFN prior and greedily select the subset with the highest winrate over the random baseline. Online, a new dataset fits only the selected configurations, so the gain comes at no extra cost. Empirically, we observe that scaling both the number of configurations and synthetic datasets has an equal scaling effect on the performance of the final subset of configurations, which we also explain theoretically. On the full BeyondArena benchmark, LightGBM with TabPrep features with our configurations ranks above TabPFN-3 and RealMLP, while being much more efficient at inference. Overall, our results establish existing PFN priors as a scalable source of meta-training data for hyperparameter selection for non-foundational tabular models.
Machine Learning (cs.LG)
arXiv:2611.05141 [cs.LG]
(or arXiv:2611.05141v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2611.05141 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 264 words
Related coverage
Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training
arXiv · Jun 5, 2026
Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
arXiv · Sep 2, 2026
The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It
arXiv · Oct 21, 2026
Efficient Hyperparameter Optimization for LLM Reinforcement Learning
arXiv · Jun 3, 2026