perspectiva aiqbenchCercetare AI

This article provides a technical comparison between Simplex and Timaeus, two organizations pursuing fundamental science approaches to AI alignment through distinct mathematical frameworks. It contrasts Singular Learning Theory's focus on weight space and training with Computational Mechanics' emphasis on activation space and inference via Bayesian updating. The piece highlights the ongoing theoretical debate regarding whether these domains are fundamentally distinct or merely different perspectives on Bayesian conditioning in large language models.

Sursă care acoperăLessWrong

Ce este aceastaLectura unui model asupra articolului și a acoperirii conexe, scrisă pentru aiqbench. Nu sunt cuvintele publicației.

Simplex vs Timaeus: Round One

LessWrong1 octombrie 20267 min de cititArticolul și perspectiva sunt în engleză.

I'm often asked about the differences and similarities between Simplex' and Timaeus' research agendas. The question is natural enough. Both focus on a 'fundamental science' approach to AI alignment. Both organizations base their research agendas on sophisticated mathematical frameworks handed down from a bearded ur-figure (Sumio Watanabe, James Crutchfield). We may posit the following correspondence Dan Murfet + Jesse Hoogland : Developmental Interpretability : Singular Learning Theory : Sumio Watanabe <-> Adam Shai + Paul Riechers : Belief-state Geometry: Computational Mechanics : James Crutchfield SLT vs CompMech Round One. Fight! Weights vs Activations DevInterp & SLT is about weight space. Belief-state geometry is more about studying activation space. Activation space is what is already being studied in MechInterp & most 'mainstream' approaches to interpretability. It is concrete and present to the senses. Weight space is much larger, more abstract, harder to measure and sample. Training vs Inference SLT is about training. CompMech is about inference. Both study Bayesian posteriors and updating. For SLT that is the Bayesian posterior on weight space - hence relevant for training. The Belief-State Geometry agenda studies the Mixed State Presentation from CompMech which describes an [idealized] version of in-context learning as the LLM doing Bayesian updating token-by-token as it reads the context. Caveat. It isn't clear that inference and learning are really fundamentally distinct. It's all just updating/conditioning in Bayesian statistics. Indeed, if one buys the Strong in-Context Learning story the difference between the training/learning of LLMs and inference may be somewhat illusory. IID vs non-IID Data Singular Learning Theory has historically been about IID data. CompMech about IID data is trivial. Caveat. Singular learning theory has been studied for non-IID data but it's fair to say its development is in its early stages. Parameterization vs Invariant structure Computational Mechanics is all about the invariant structure. A fundamental objects is the epsilon machine (and its mixed-state presentation) which is obtained by quotienting out all the details of the past (read: the context) that do not matter for predictions of the future (read: continuation) Singular learning theory would be trivial if not for the weight-parameterization of the loss function. The important bit is that the loss landscape of loss functions on neural-network architectures are singular. That is a feature of the specific parameterization of the underlying function/ probability distribution by the neural network architecture. Asymptotic vs Exact Singular Learning Theory's fundamental theorem is Watanabe's Free Energy formula which describes the Bayesian posterior in the large data limit. Computational Mechanics can equally be applied to large as well as small data regimes. Mechanism vs Behaviourial Belief-State Geometry posits a model for the underlying internal mechanism of LLMs. The devinterp agenda flags when important structure forms. SLT flags structure forming. CompMech has a mechanistic story of what that structure is. Bottom-up vs Top-down interpretability Computational mechanics has a ground-level truth of the data process. By contrast, outside of a few simple examples (Deep Linear Networks, shallow ReLU neural networks) the central quantity of SLT the famed learning coefficient mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { wil

Round One. Fight!

DevInterp & SLT is about weight space. Belief-state geometry is more about studying activation space.

Activation space is what is already being studied in MechInterp & most 'mainstream' approaches to interpretability. It is concrete and present to the senses. Weight space is much larger, more abstract, harder to measure and sample.

SLT is about training. CompMech is about inference.

Both study Bayesian posteriors and updating. For SLT that is the Bayesian posterior on weight space - hence relevant for training. The Belief-State Geometry agenda studies the Mixed State Presentation from CompMech which describes an [idealized] version of in-context learning as the LLM doing Bayesian updating token-by-token as it reads the context.

Caveat. It isn't clear that inference and learning are really fundamentally distinct. It's all just updating/conditioning in Bayesian statistics. Indeed, if one buys the Strong in-Context Learning story the difference between the training/learning of LLMs and inference may be somewhat illusory.

Singular Learning Theory has historically been about IID data. CompMech about IID data is trivial.

Caveat. Singular learning theory has been studied for non-IID data but it's fair to say its development is in its early stages.

Computational Mechanics is all about the invariant structure. A fundamental objects is the epsilon machine (and its mixed-state presentation) which is obtained by quotienting out all the details of the past (read: the context) that do not matter for predictions of the future (read: continuation)

Singular learning theory would be trivial if not for the weight-parameterization of the loss function. The important bit is that the loss landscape of loss functions on neural-network architectures are singular. That is a feature of the specific parameterization of the underlying function/ probability distribution by the neural network architecture.

Singular Learning Theory's fundamental theorem is Watanabe's Free Energy formula which describes the Bayesian posterior in the large data limit. Computational Mechanics can equally be applied to large as well as small data regimes.

Belief-State Geometry posits a model for the underlying internal mechanism of LLMs. The devinterp agenda flags when important structure forms. SLT flags structure forming. CompMech has a mechanistic story of what that structure is.

Computational mechanics has a ground-level truth of the data process. By contrast, outside of a few simple examples (Deep Linear Networks, shallow ReLU neural networks) the central quantity of SLT the famed learning coefficient is not known analytically. The learning coefficient is estimated but we know this estimator can be very wrong, both empirically and on first principles.[1]

The belief-state geometry agenda has historically focused on understanding mechanistically small toy models. Perhaps its chief challenge is porting that understanding to large models for which we know the language of HMMs must be inadequate. Some relevant work here: Simplex paper on factored representations. Crutchfield's visionary[2] paper on calculi of emergence.

By contrast, Timaeus moved to large scale experiments quickly. Rather than understanding mechanistically every single detail the point is to understand what features are truly important (as flagged by their learning-theoretic behaviour).

Physics vs Math?

This one is less clear. Daniel Murfet and Sumio Watanabe are mathematicians while Paul Riechers and James Crutchfield are physicists.

However, SLT has close connections with statistical mechanics and Daniel Murfet has an extensive background and track record in mathematical physics. And computational mechanics is unusual within the larger space topics which it overlaps with (quantum physics, complex systems, chaos theory, etc) for its emphasis on precise mathematical definitions and careful conceptual grounding.

I pronounce: Simplex and Timaeus are neither competitors nor natural allies. Their historical domains of study are largely orthogonal. That may change as they start to invade each other's stomping ground.

On the whole, I regard this as a very natural and healthy state. A time-tested way to make progress on very hard problems is breaking them up into separate, mostly orthogonal phenomena and study each phenomena carefully and systematically before. Inshallah, this ultimately paves the way for a paradigm-shifting synergy.

^cf Dmitry Vaintrob's Tanh network example.

cf Dmitry Vaintrob's Tanh network example.

Text integral sub licența deschisă a publicației · 714 cuvinte

Citește pe LessWrong ↗Întreabă despre acest articol