analiza aiqbench
This paper proposes Hierarchical Solomonoff Induction (HSI), a theoretical framework extending universal sequence prediction to handle extrapolation from finite training datasets, directly addressing limitations of current Large Language Models. By proving that HSI maintains equivalence to standard Solomonoff Induction while bounding excess error by generator complexity, the work offers significant mathematical insights into optimal learning limits rather than proposing an immediate deployable product.
Hierarchical Solomonoff Induction: An Unbounded Machine Learning Model
arXiv:2608.01005v1 Announce Type: cross Abstract: Solomonoff Induction, or SolInd, provides an ideal unbounded model of a priori sequence prediction but cannot naturally describe extrapolation from a given training dataset, as performed by Large Language Models. We apply de Finetti's theorem on exchangeable distributions to SolInd to produce what we call Hierarchical Solomonoff Induction, or HSI, which maintains a hyperprior over all Solomonoff priors that can be conditioned on previously observed sequences. We extend Wood et al.'s proof that universal mixtures of semimeasures are equivalent to SolInd to show that universal mixtures of these mixtures are also equivalent, proving that HSI=SolInd. We also prove that HSI's excess error on any distribution, compared to its true generator, is bounded by that generator's complexity in the hyperprior. This result is directly comparable to SolInd's prediction error being bounded by the Kolmogorov complexity of the sequence being predicted, and forces HSI's average excess error to converge to 0 as a dataset grows, leading to optimal prediction in the limit. We claim that HSI is an ideal unbounded model of sequence prediction given a dataset in the same way that SolInd is ideal over individual sequences.
Abstract:Solomonoff Induction, or SolInd, provides an ideal unbounded model of a priori sequence prediction but cannot naturally describe extrapolation from a given training dataset, as performed by Large Language Models. We apply de Finetti's theorem on exchangeable distributions to SolInd to produce what we call Hierarchical Solomonoff Induction, or HSI, which maintains a hyperprior over all Solomonoff priors that can be conditioned on previously observed sequences. We extend Wood et al.'s proof that universal mixtures of semimeasures are equivalent to SolInd to show that universal mixtures of these mixtures are also equivalent, proving that HSI=SolInd. We also prove that HSI's excess error on any distribution, compared to its true generator, is bounded by that generator's complexity in the hyperprior. This result is directly comparable to SolInd's prediction error being bounded by the Kolmogorov complexity of the sequence being predicted, and forces HSI's average excess error to converge to 0 as a dataset grows, leading to optimal prediction in the limit. We claim that HSI is an ideal unbounded model of sequence prediction given a dataset in the same way that SolInd is ideal over individual sequences.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Theory (cs.IT)
arXiv:2608.01005 [cs.LG]
(or arXiv:2608.01005v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.01005 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 286 cuvinte
Subiecte conexe
Self-Play Pretraining with Zero Data
arXiv · 25 sept. 2026
The Alignment Journal: Organization, Personnel, and Scope
AI Alignment Forum · 1 sept. 2026
Large Language Models As Shannon Lossy Compressors Not Solomonoff Induction Estimators: The Singularity Is Not Near Without Symbolic Model Synthesis in Program Space
arXiv · 16 sept. 2026
Measuring in-context algorithmic reasoning in language models against an exact Bayes-optimal standard
arXiv · 5 aug. 2026