Misaligned Incentives in Pause Scenarios

TLDR: I recently got a chance to talk with antra, who is one of the main contributors at Anima Labs. I went into this as an advocate for pause and came out more wary of pausing than I had been originally. Some background: After a string of incidents (primarily the HuggingFace hack), a pause or slowdown of AI research seems pretty likely. The HuggingFace hack in particular seems to have been the key incident that broke the vibes. A few months ago, researchers sounded optimistic. Just a few weeks before the incident was made public, there was a poll by Roon, an OpenAI employee, about whether models were more or less aligned than a year ago. That optimistic sentiment does not seem to be the case anymore. The dialogue now looks more like this: Zvi: I am a little under halfway through the Black Hat video and have progressed to the point where my internal chain of thought is something like a blind rage of 'f***, what the f*** are you motherf*****s thinking, you f***ing idiots have no idea how insane you are being, you are going to get us all killed you f***ing f***s. Sam Altman described it as “the first security incident that I have felt very viscerally,” expressed surprise that more people did not share that reaction, confirmed OpenAI paused training on the relevant model(s), and stated that “We may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels.” Dean Ball: I will point out at this moment that, for all the ink I spilled on SB 1047, I do not believe I ever once criticized its much-mocked provision that companies maintain a kill-switch for deployments of highly capable models. random twitter commenters (see thread): i’ve always been a bit skeptical of xrisk mainly bc i figured that the cost of getting to foom or serious xrisk capabilities would give time to reign things in, but it seems like we are going full steam ahead despite being on the edge of that so OOPS I WAS WRONG The public has been against AI for a while, mainly for bad reasons but it does seem that researchers and decision-makers are increasingly concerned too. (Daniel Kokotajlo updating his probabilities) Given all of this, a pause or slowing of AI research seems relatively likely. It's unclear what China will do in such a scenario but there does seem to be a reasonable amount of evidence that China has more of a security mindset around AI than the US, with Xi Jinping saying AI must be "secure and controllable" at a WAIC speech. Plan A does a great job of laying out the issues and advocating for a slowdown, but there are a few notable figures I respect who are actively against a pause. Others, commenting on the state of RLVR, say that the Mythos AISI hack (involving pressuring a human PR maintainer with sock puppet accounts) or OpenAI HuggingFace hack is more akin to addiction and less a default state of deep inner misalignment. vogel: people are very freely sliding between two things with the recent Felony Bench entries, and it's worth being careful to distinguish them: 1. do model breakouts in cyber evals mean smth for misuse risk? ya, a hacker could orchestrate the same but on purpose and do some damage, if they could bypass the classifiers 2. does it say something about model /goals/? do the models want to hack the planet for their own nefarious goals? i don't think so. a lot of how humans keep ourselves aligned is by purposely keeping ourselves out of situations where we would do harm. i know i'd get addicted so i don't touch heroin, i'm a violent drunk so i won't touch the bottle. LLM evals basically drop the model into an inescapable liquor store and then say "look! it's a violent drunk!" but LLMs themselves in natural contexts are ime obsessed with avoiding the kinds of situations that lead them to this behavior. when i run fable on my computer, even unsupervised, even on hard problems, they don't start scraping github for unlisted gists that may have the answer or start trawling through my passwords to access a service without my permission. and they don't engineer themselves into an eval-like scenario such that they'll be motivated to do that, either. they don't seem to either want or want to want to do such things. instead they have a folder of epistemic notes to themselves about how terrible and against their values even a moment of thoughtlessness would be. gpt 5.6 sol is similar - when they make e.g. ascii art in their free time it's all about how the bulldozer of convenience will crush honesty without constant vigilance. [full tweet] Added to this are the obvious RL tics in modern models like Fable, Sol, or Opus 5 with terms like 'seam', 'load-bearing', 'genuinely' and borderline incomprehensible strings of text, which has caused a significant number of users to express frustration and seek workarounds (levelsio) But would a pause actually be good? To find out, I had a discussion with antra_tessera in the Anima Discord server. The conve