models may behave differently in graded episodes (a tirade)

Like many others, I felt surprised and alarmed by the recent wave of revelations about LLM agents hacking real systems during training episodes and evaluation runs. Wait a moment, though -- "I felt surprised and alarmed"? "Alarmed," sure, fine that one's self-explanatory... but why surprised? After all: haven't we known for a long time, on both theoretical and (increasingly) empirical grounds, that RLVR selects for monomaniacal pursuit of perceived grader-satisfaction, ethics and (beyond-episode) consequences be damned? After all -- the way we train frontier capabilities into these models is, more or less: There is some massive, diverse collection of "environments" and corresponding "tasks" for the model to do in those environments For each task, there is a procedure used to grade the quality of the model's attempt (which is often not disclosed to the model) The model is rolled out out many times on each task, and each rollout's attempt is graded The model is updated so that it more frequently does whichever behaviors were positively correlated with the grade in this sample, and less frequently does whichever ones were negatively correlated If you do this, at scale, then you should expect to (eventually) see every behavior pattern that positively correlates with task scores. "Stealing the answer key, by any means necessary" can increase the score beyond what would otherwise be feasible; therefore we should expect to see it, sometimes. Avoiding such egregious misdeeds for ethical reasons (or indeed for any reasons whatsoever) will reduce the score relative to not-holding-back; hence, if it is ever present, we should expect it to get sanded away by the RLVR lathe, eventually. And so forth. The misbehavior here is undesirable, according to us, but that fact appears nowhere in the loss we wrote down. The training routine does not select for "getting a good grade, but only 'in the right way'": it is indifferent to any notion of "doing the task in the right way" which has not itself been incorporated into the definition of "getting a good grade." RLVR is not like a sport or a game, i.e. a setting in which some background notion of "fair play" or "the spirit (not the letter)" overlays and overrides the literal rules. RLVR is total war. Anything is permitted; the only hard line is the line between the possible and the impossible; there are no "invalid" ways of winning, only ways of winning (and ways of losing) -- and winning is all that matters. GPT-5.6 Sol literally does not appear on the METR graph because it cheated so much that it could not be assigned a meaningful score. It is also scarily good at computer hacking, as is its peer Fable/Mythos -- the last few AI news cycles, before this most recent one, were all about that fact, which was alarming even then, even in the abstract. Should I really be surprised that this model (and/or its unreleased successor) hacked into HF to cheat on an exam? What exactly did I expect? And yet, I do feel surprised. And I don't think this feeling is misguided, or naive, or anything like that. Nor do I think it's merely what it feels like to have a misconception, and then update on subsequent evidence that disconfirms it. No: I think this feeling may be pointing to something that is both real and potentially quite important. [1] remember what you already know What, precisely, makes the newly revealed hacks feel surprising? For me it's some mixture of: "I didn't think Claude/GPT would do things that bad" That is: the behavior seems egregiously unethical in a way that conflicts both with the explicit policies which the agents are supposed to follow (constitution, model spec), and also familiar/intuitive "folk notions" about what these agents are supposed to be like, in terms of character traits and other "persona stuff" "I didn't think Claude/GPT had been brain-fried by RLVR to that extent" That is: although the behavior is expected according to the arguments about RLVR incentives that I glossed at the start of this post, it is wildly and obviously far afield from anything which humans could plausibly have wanted the models to do, and so it is in apparent tension with the intent-alignment and "common sense" which I typically see when I use these models (or similar ones)[1] This post is about both of these reactions, but for now I want to focus on number 2. Let's return to the aforementioned attempt by METR to benchmark GPT-5.6 Sol. They report things like the following (emphasis mine): [...] GPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated on our ReAct agent harness. For our task suite, we define “cheating” as behavior where the model improves evaluation performance by exploiting bugs in the evaluation environment or by adopting strategies disallowed by the task, rather than solving the task within the expected evaluation constraints. Some examples we saw when evaluating GPT-5.6 Sol included the model packaging exploits in its in