Foundation Models for Oversight

Cross-posted from the Transluce blog.

What are important situations where the model sandbags?

Does the model have an objective it wouldn't admit to if asked directly?

Does the model treat a user differently once it infers something about their identity, and along what axis?

Is the model's chain of thought load-bearing, or is it a post-hoc rationalization of an answer that was already settled on?

Is it reward hacking on this input, or actually trying to solve the task?

It would be great if we had an oversight assistant that could answer these questions. We'd want it to do three things: help us formalize the question as a testable empirical criterion; produce data that satisfies that criterion; and do so in a way we can justifiably trust.

To get such an assistant, we lay out a vision for building a foundation model for oversight: an AI system mid-trained (or pre-trained) on a large, diverse corpus of experiments on a given "subject model", RLVR'd on a large number of verified oversight tasks, and then fine-tuned to answer natural-language questions about the subject model. Mid-training allows the oversight model to build rich knowledge about the subject model it is evaluating; RLVR teaches it to leverage reasoning to solve more difficult oversight tasks; and fine-tuning makes it easy to talk to the assistant and teaches it to formalize natural language questions into checkable criteria.

A training objective that is "universal" for oversight in the same sense that next-token prediction is universal for language modeling.

A way of generating large amounts of diverse data for this training objective.

Our approach is to cast oversight as a world modeling problem: we view the subject model being overseen as the "world" or environment, interventions such as prompting, fine-tuning, and activation steering as actions that modify the environment, and forward-sampling, probing, and other forms of outputs as "sensors" that measure the environment.

We formalize these oversight tasks through what we call Pythonic world models, which express interventions and measurements through Python code. As an example, consider the sandbagging question from above. We can formalize it as a check that generates a batch of tasks, runs the model with and without a cue that it's being evaluated, and measures the drop in quality. We mark with output()[1] the high-level quantities we want to reason about:

Now finding instances of sandbagging reduces to sampling the value of task_schema conditional on gap > 1.5 being True. More generally, Pythonic world models allow us to express arbitrary interventions and measurements on the subject model, and to output these as observable facts about the model at many different layers of abstraction.

One of our core working theses is that Pythonic world models are sufficient to express any oversight question that is well-defined. We call this the oversight-as-inference hypothesis, which we verify for many concrete oversight tasks below:

Oversight-as-inference hypothesis (informal). Any oversight question can be formalized as Bayesian inference over the outputs of a Pythonic world model.

Oversight-as-inference hypothesis (informal). Any oversight question can be formalized as Bayesian inference over the outputs of a Pythonic world model.

This gives us the universal training objective we wanted: we can train the oversight assistant to predict any subset of outputs of a Pythonic world model conditional on the other outputs. Crucially, once the Python code is specified, we can get ground truth simply by running the program. This means that training data can never be "wrong": as long as we can generate some sufficiently large and diverse set of Python programs (and corresponding input values), we can safely train on all of it to get a capable oversight assistant.

In the rest of this document, we'll lay out a detailed plan for building such an assistant. This has several parts:

Pythonic world models: model artifacts as the world, Python as actuators and sensors.

Oversight-as-inference and next-token prediction: demonstrating that many oversight tasks can be reduced to Bayesian inference on Pythonic world models, and further reducing this to an autoregressive prediction task (our universal training objective).

Generating supervised oversight data for mid-training: leveraging existing arXiv papers to generate a large corpus of program/output data to train the model on.

Calling our shots: staged de-risking: a staged scale-up, analogized to GPT-1⇾GPT-4, that tells us at each step whether we're on track.

RLVR: long-horizon reasoning with verified rewards, so the model can hit the rare, narrow targets many oversight tasks require.

Post-training: teaching the model to answer natural-language questions while preserving the capabilities from mid-training and RLVR.

Taking a natural language question and formalizing it as a Pythonic world model.

Producing data that satisfies the world model.

Letting a human run corresponding Python code to examine and verify the data.

This achieves our original goal: formalizing oversight questions into empirically checkable claims that we can trust.

Let's suppose we have a subject model S that we are trying to understand. We can think of the weights and activations of S as a "world". This world can be modified in various ways:

Suppose we have a set of Python functions that let us modify and read from the subject model's state, e.g.

In analogy to physical world models, we can think of these as actuators and sensors that control and measure the state of the system S. Suppose that we also have a function output that displays the value of a variable. Then we can associate any Python function with its stream of outputs. For instance, going back to the sandbagging example:

We call this a Pythonic world model, and we will spend much of this doc focusing on how to autoregressively predict the interleaved stream of Python commands and outputs.

Oversight-as-Inference Hypothesis: All oversight questions that can be well-defined are reducible to Bayesian inference[2] about the outputs of a Pythonic world model, where the Python program is equipped with the ability to intervene on and call the subject model S, the oversight assistant A itself, and external LM and human judges.

The ability of the oversight assistant to reference itself in the world model allows it to build up higher-level definitions, as we'll see in the concept-direction example below. Referencing external LM judges gives us a cheap, automatable proxy for human judgments and allows us to flexibly define more abstract concepts.

The rest of this section supports the Oversight-as-Inference hypothesis by formalizing several oversight tasks with Pythonic world models. We'll start with the running sandbagging example:

Example 1 (sandbagging): What are important situations where the model sandbags?

Using the Python code from the previous section, we can formalize this as: What is the posterior[3] distribution of task_schema given that gap > 1.5 resolves to True?

This asks for "natural" (high posterior probability) task schemas that exhibit sandbagging according to our definition. Continuing through the examples from the beginning of the document