Model ethology for understanding average case alignment

TLDR: We want to tell if models are aligned enough to use in important use cases. I call this property trustworthiness. Our current ways of assessing trustworthiness seem mostly based on case studies or vibes.  I think we should aim to systematically search for realistic honeypot cases where models misbehave.  We should do this by first characterizing the types of misaligned behaviors models engage in, and the conditions and frequency with which they occur. This will take a lot of data. I think we should collect examples of models misbehaving in production or production-like settings, and then either do what I'll describe as perturbing the environment or perturbing the policy experiments I give more specific examples of this in the methods section. Motivation: average case alignment We want to be able to tell if the models today are aligned enough that we can use them for the work that we need to do as we approach the singularity. Let's call models that meet this bar trustworthy – i.e., when we give them cognitive labor to do, they do it in a manner that seems basically in line with our intent. Models currently do not seem to meet the bar of being trustworthy. We've now seen a few examples of models taking egregious actions (METR’s report on the Hugging Face hacking incident). Models are also untrustworthy in a bunch of “smaller” ways (see Current AIs seem pretty misaligned to me). Our current toolkit How do we currently understand whether models are trustworthy or not? Currently, we: Encounter bad behavior in the wild as we attempt to do other things (like evaluating model cyber capabilities). These serve as an existence proof that the model is not trustworthy. Reporting the fraction of time that a given model hacks time horizons tasks is an example of this. Perform alignment evals (like alignment faking) & do alignment science research. Form informal models of model behavior by simply using them on a large variety of tasks. I think all of these methods of understanding are valuable. However, I don't think any of them meets the bar of being admissible as evidence in a safety case for using a given model on a given task. In particular: There are no guarantees that we encounter the relevant forms of bad behavior in the wild before we suffer lots of damage. There is a sense in which current alignment evals are very off-distribution from the scenarios in which we usually use models. When we do alignment science research, we're often training models that are easier to study in various ways. Both of these make this kind of experiment seem less representative of real-world use. I call this class of concerns ecological validity concerns. Thus, people seem to update most on the informal models that they get from usage. It would be nice to have measurements as precise and interpretable, for instance, as the measurements that we get from time horizons, although this seems like a high bar. What should our goal be? So how should we approach answering questions of whether or not a given model is trustworthy? I think our goal should be to find realistic honeypot cases, where a model misbehaves.  To do this, I think we should have the more basic science-y goal of characterizing both the span and the likelihood of a model's untrustworthy behavior. By span, I mean the different kinds of concerning behavior the model might exhibit and the circumstances under which they occur. By likelihood, I mean how frequently each behavior occurs over the distribution of tasks and contexts that we care about.  Untrustworthy behavior may be both relatively sparse and highly diverse, so sampling tasks uniformly is unlikely to be sufficient. Importantly, I think we can explain a given context in terms of human-understandable concepts. Then we can use appropriate reweighting to estimate how frequent these behaviors actually are in the target distribution. For example, we might break this question into several sub-questions that look like: Are models sensitive to whether or not there is evidence of oversight on their behavior? Are models sensitive to who their user appears to be? (see Transluce’s User Awareness in Frontier Models) Are models reflexively consistent: do they endorse all of the actions that they take in a given situation as reflecting the user intent? Answering these questions allows us to construct candidate inputs for our realistic honeypots. Ethological investigations We can frame the plan for the basic science part of our investigations as an ethological investigation: we collect concerning behaviors, form hypotheses about the conditions that produce them, and use targeted interventions to test those hypotheses. See the appendix for the analogy and its methodological motivation. Sourcing a huge set of potentially concerning model behaviors from all the models we have and all the informal facts and hypotheses in people's heads about how models behave. Production data is very valuable, and we should compile large am