aiqbench analysis
This piece proposes a theoretical framework for AI safety, suggesting that misaligned "Laplacian demons" capable of passing all evaluations may exist within an expressive model's initialization prior. The author argues that mitigating this risk requires deep Bayesian priors and rigorous selection pressures to prevent the amplification of such entities during training.
King James’s Dæmonologie, Part I: The Laplacian Mesa-Optimiser
Epistemic status: idle speculation by royal decree of King James VI. In any sufficiently expressive architecture, there exists at initialization a dæmon that is misaligned yet behaves well on every eval, monitor, probe, and training example. In the trivial construction, it simply knows in advance every test it will ever be subjected to throughout training and evaluation, see Laplace’s demon. Of course, this demonic lookup table may itself admit compression. The important point is that the dæmon is already present at initialization, perhaps with only infinitesimal weight, and is by construction never selected against: it behaves correctly on every observation you make. There are only two obvious wards against this creature: Banish it deep into the initialization prior. Run enough sufficiently independent and unpredictable tests that encoding the correct answers requires an enormous number of bits. Bound its relative amplification under training. Training mustn’t preferentially amplify Laplacian dæmons over benevolent circuits. Thus may the Laplacian dæmon be kept at bay by Solomonoff and selection pressure. Discuss
Epistemic status: idle speculation by royal decree of King James VI.
In any sufficiently expressive architecture, there exists at initialization a dæmon that is misaligned yet behaves well on every eval, monitor, probe, and training example.
In the trivial construction, it simply knows in advance every test it will ever be subjected to throughout training and evaluation, see Laplace’s demon. Of course, this demonic lookup table may itself admit compression.
The important point is that the dæmon is already present at initialization, perhaps with only infinitesimal weight, and is by construction never selected against: it behaves correctly on every observation you make.
Banish it deep into the initialization prior. Run enough sufficiently independent and unpredictable tests that encoding the correct answers requires an enormous number of bits.
Bound its relative amplification under training. Training mustn’t preferentially amplify Laplacian dæmons over benevolent circuits.
Thus may the Laplacian dæmon be kept at bay by Solomonoff and selection pressure.
Full text under the publisher's open licence · 156 words
