aiqbench analysis
This article presents theoretical findings on Bayesian neural networks, establishing that specific weight priors induce circuit-like structures which yield sample complexity bounds dependent on circuit width. The research highlights critical trade-offs between overparametrization and robustness assumptions required to avoid exponential dependence on network depth. These results advance the mathematical understanding of generalization in deep learning models.
Circuit priors in NN-bayes: more results
We're posting lots of papers written with GPT-6 Astra on generalization properties of bayesian neural net learning. [1] The following picture emerges: For various reasonable priors on MLP weights, [2] the induced prior on functions is a circuit prior: for fixed depth, the ordering of functions by negative log prior is roughly their ordering by minimal circuit width. From a quantitative version of this statement, one gets the following sample complexity bound: if the function mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c1D436.TEX-I::before { padding: 0.705em 0.76em 0.022em 0; content: "C"; } mjx-c.mjx-c70::before { padding: 0.442em 0.556em 0.194em 0; content: "p"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c1D456.TEX-I::before { padding: 0.661em 0.345em 0.011em 0; content: "i"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D700.TEX-I::before { padding: 0.452em 0.466em 0.022em 0; content: "\3B5"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c78::before { padding: 0.431em 0.528em 0 0; content: "x"; } mjx-c.mjx-c398::before { padding: 0.705em 0.778em 0.022em 0; content: "\398"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] {
We're posting lots of papers written with GPT-6 Astra on generalization properties of bayesian neural net learning. [1] The following picture emerges:
For various reasonable priors on MLP weights, [2] the induced prior on functions is a circuit prior: for fixed depth, the ordering of functions by negative log prior is roughly their ordering by minimal circuit width. From a quantitative version of this statement, one gets the following sample complexity bound: if the function is implemented by a circuit of width , then conditioning on train data points is sufficient to get test accuracy. [3] These results hold as long as is realizable; in particular, for arbitrarily overparametrized NN-bayes. Probably the main caveat is that these bounds have exponential dependence in the depth of the circuit implementing . One can get only polynomial dependence in depth by restricting the prior to only neural nets satisfying a robustness assumption. [4]
These results hold as long as is realizable; in particular, for arbitrarily overparametrized NN-bayes.
Probably the main caveat is that these bounds have exponential dependence in the depth of the circuit implementing . One can get only polynomial dependence in depth by restricting the prior to only neural nets satisfying a robustness assumption. [4]
For various other reasonable priors on MLP weights, [5] , one gets bad sample complexity when very overparametrized: when learning some functions , even after seeing train data points, the learner is still likely to have at least some constant test error rate.
GPT-6 Astra has produced Lean formalizations corresponding to each of the 5 papers individually referenced above. [6] The folder contains 30+ more manuscripts which have not been formalized yet [7] ; our guess is that each manuscript is correct [8] .
If you want to understand this research program, we recommend starting from these slides to understand the setup and some key ideas, and then reading the FAQ to get a sense of the current state of the agenda, including its many open problems.
There is a host of possible conditions here; there are open research directions of understanding how reasonable our candidate conditions are, coming up with more natural alternative conditions, or arguing that there isn't any very natural condition. For some cases, it is open if a robustness assumption is needed at all. ↩︎
There is a host of possible conditions here; there are open research directions of understanding how reasonable our candidate conditions are, coming up with more natural alternative conditions, or arguing that there isn't any very natural condition. For some cases, it is open if a robustness assumption is needed at all. ↩︎
Opus 5.5 reviewed these formalizations and said all theorem statements were formalized correctly. ↩︎
Opus 5.5 reviewed these formalizations and said all theorem statements were formalized correctly. ↩︎
Full text under the publisher's open licence · 463 words