Structured Priors: Towards an algebra of component priors for prior-data fitted networks

Abstract:Prior-data fitted networks are typically trained using synthetic data generated by a single, manually constructed generator. In cases where such a generator combines multiple components, e.g., in TabPFN, Mitra, or TabICLv2, this is achieved through a mixture using manually specified weights, the impact of which on the network only becomes apparent after pre-training. Instead of such a hand-built generator, this work proposes a structured prior: a small set of named component priors, linked by operators whose algebraic properties are formally proven and whose influence on the trained network has been validated. Within a measure-theoretic framework, the operators of mixture, convolution, amplitude scaling, local transform, and projection are defined. Their governing laws are proven. These operators are implemented in the macl2l software package. Experiments demonstrate that the implemented generators reproduce the identities within the resolution of a permutation test. Furthermore, it is shown that trained networks obey the distributive law; that a mixed prior composed of small components performs as well as or better than any individual component in seven out of eight open regression benchmarks; and that on two of these tables a structured prior reaches, with a ninth of the data budget, an error that a generic random-function prior does not reach even with the full budget. A theorem on model averaging demonstrates that networks trained on individual components constitute the Bayes predictor for the mixture, provided that the component networks are themselves the Bayes predictors for their respective priors. However, the pre-registered test of this averaged predictor with trained component networks failed, because the networks identify the generating component too rarely. The paper proposes a research program focused on the algebra of structured priors for prior-data fitted networks.

Machine Learning (cs.LG)

arXiv:2611.04056 [cs.LG]

(or arXiv:2611.04056v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2611.04056 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.