Core Assumptions of ELK

In this post I do a brief analysis of the core assumptions of ELK. Context The Eliciting Latent Knowledge problem, for the unfamiliar: Suppose we train a model to predict what the future will look like according to cameras and other sensors. We then use planning algorithms to find a sequence of actions that lead to predicted futures that look good to us. But some action sequences could tamper with the cameras so they show happy humans regardless of what’s really happening. More generally, some futures look great on camera but are actually catastrophically bad. In these cases, the prediction model "knows" facts (like "the camera was tampered with") that are not visible on camera but would change our evaluation of the predicted future if we learned them. How can we train this model to report its latent knowledge of off-screen events? The most analyzed version of the problem is a toy scenario with a diamond in the vault: Imagine you are developing an AI to control a state-of-the-art security system intended to protect a diamond from theft. The security system, the SmartVault, is a building with a vast array of sensors and actuators which can be combined in complicated ways to detect and stop even very sophisticated robbery attempts. (...) The SmartVault can execute plans sufficiently sophisticated that humans can’t really know if the diamond is safe or merely appears safe. Whatever complicated hard-to-follow sequence of actions the search procedure found might actually have replaced the diamond with a fake, or tampered with the camera (...) So we have humans with their human ontology, the Predictor with an arbitrarily complicated/alien ontology, and we're training a Reporter (intended to interpret the Predictor). We assume the Predictor "knows" whether the diamond is in the vault. We want to extract that information. Good Reporters - direct translators - tell if the diamond is really in the vault; bad Reporters - human simulators - tell if humans will believe that the diamond is in the vault. We assume the Predictor is inner-aligned to myopically making predictions; it doesn't actively try to kill or mislead humans. We also assume that any combination of sensors can be fooled/hacked. ARC's working definition of knowledge: To show that M [a given model] doesn’t solve ELK, we want to construct a counterexample where M(x) provides answer A to question Q, instead of alternative answer A′, despite all of the following features which together strongly suggest that M(x) knew that A′ was the better answer: Question Q is unambiguously about something “out there” in the real world, call it Z. The interpretation is unambiguous and straightforward in the context of input x and also for all of the relevant training examples. A and A′ are very different answers to question Q. For example, if they are estimates of a quantity then the difference is large relative to uncertainty about the quantity or ambiguity about its definition. That real world referent Z has observable effects and the human approximately understands those effects (though there may be other things that also affect observations which the human doesn’t understand) The AI understands everything relevant about how the human uses language. The AI can do tasks that require predicting human utterances about the relevant concepts, including discussions of hypothetical cases and about how Z affects observations. M would answer Q correctly (as judged by the human) in cases where the human was able to estimate Z correctly (if doing so led to M getting a lower loss). The referent Z is also relevant to minimizing the loss function ℒ. That is, there is a coherent sense in which the optimal behavior “depends on” Z, and the relative loss of different outputs would be very different if Z “had been different.” There is a feature of the computation done by the AI which is robustly correlated with Z, and for which that correlation is causally responsible for M achieving a lower loss. On input x in particular, M(x) produces outputs that receive a significantly lower loss in the real world, where A′ is the right answer to Q, than in some kind of counterfactual where A is a better answer. This is in virtue of the same feature of M’s computation that is robustly correlated with Z. We aren’t offering these criteria as necessary for “knowledge”—we could imagine a breaker proposing a counterexample where all of these properties are satisfied but where intuitively M didn’t really know that A′ was a better answer. In that case the builder will try to make a convincing argument to that effect. We don’t really expect to get a formal definition of the problem until it is solved. Until then, the ultimate arbiter is human judgment. Although this situation is unsatisfying, we want to emphasize that for existing approaches and counterexamples there isn’t much ambiguity coming from the definition of “knowledge”---the approaches just unambiguously fail. Philosophical Analysis On one han