What pacing the frontier means for China

Dario's recent post on pacing the frontier mentioned how slowing down China more would help with international coordination. It seemed counter-intuitive, surely sabotaging their efforts would create badwill, but it may give the US more leverage to negotiate. Would China be hyper-rational about it, or would hurt feelings make coordination tougher? Even today, looking at smart people responding so differently to AI risk seems very counter-intuitive. Something related to incentives and wishful thinking. Here are some quick explorations with some numbers. The objective function Let us define reward R as the objective function that each person tries to maximize. Let's assume we maximize the EV of R. I can think of 2 reasons R is different for different people : They want to work towards different things. They want to work towards the same thing but reason about likelihoods differently. One of them is likely more correct and the other is more irrational in their estimates. An example of 1 could be someone mission driven (net good including others > wealth) vs someone chasing money or status. An example of 2 could be someone engaging in wishful thinking or denial or not being well informed enough. For example, the likelihood estimates in ai-2040.com are probably much more well informed than by someone with limited context. Can someone who cares about the world be corrupted by incentives? Dario and Sam may have a very similar final goal, something along the lines of "Humanity that thrives in an age of abundance in a democratic world." However incentives shape how they calculate their likelihood estimates. For example : If OpenAI has it's IPO coming up, and not enough people inside OpenAI voice concerns about AI safety, the likelihood in Sam's head about AI going badly would be lower. A super-intelligent, hyper-rational human would not be prey to this but we're not super-intelligent and hyper-rational. Similarly a lot of VCs who have incentives for an open AI ecosystem, want to see startups they are investing in thrive. They may be engaging in wishful thinking that reduces their likelihood estimates of AI going badly. Or they may feel averse to reading the relevant AI safety literature. However organizations and nations may be much more rational than individuals, owing to the diversity of perspectives and experts involved. The US China situation If the US tries to slow China down, will badwill reduce the chance that we coordinate? There are examples where emotional reasoning led to bad decisions by states. Japan, 1941. After the US oil embargo, Japan's planners estimated it would love a long war with the US. It still attacked Pearl Harbor, because accepting the embargo's terms read as national humiliation. Austria-Hungary, 1914. The assassination of Franz Ferdinand was evidence about a Serbian nationalist network. But it turned into a war to punish Serbia. The US after 9/11, Iraq 2003. The 9/11 attack was evidence about al-Qaeda. It soon became the lens every threat was seen through, and it led to a lot of anti-Iraq sentiment. But it seems in this case, that having some chip export controls will not be seen by China as humiliating or deserving of military retaliation. So let's model China as a rational player. The incentive matrices Each side has two moves. Trust means following the agreement. Hedge means secretly go faster. Hedging can mean going faster without racing at full speed to catch up. However if one side races, the other will too. Dario's post has two kinds of agreements: Level 3 : Some kind of pacing agreement to reduce speed of progress. Level 4 : A full pacing, or a pause. Where we pace conservatively. Reward Equations Let's model normalized rewards as : R_US = 𝟙[humans in control] × 𝟙[liberal order survives] × (0.7 × US power + 0.3 × abundance for it's people) R_China = 𝟙[humans in control] × 𝟙[regime survives] × (0.6 × China power + 0.4 × abundance for it's people) With 𝟙[humans in control] meaning 1 if humans are in control and 0 if not. Fixed assumptions: Abundance for people is 0.9 in every world where humans survive, for both countries. Inspired from estimates in AI 2040, estimated P(humans stay in control) : 85% if both pause. 75% if both pace under a conservative speed limit. 60% if one side secretly goes faster. 50% if both do, or if there is no deal. 30% if anyone races. Power is 60/40 today and stays there under an agreement. Cheating under a speed limit gets you 75/25. Cheating under a pause gets you 1/0. If someone wins outright, the loser's system survives only if the winner holds back. China thinks the US would hold back 40% of the time. The US thinks China would 30% of the time. Otherwise each system survives 95%. A cheater gets caught 10% of the time without verification, 90% with verification measures in the deal. Here are the reward EVs in different situations: Outcome Cell R_US R_China Both pause, honest (85%, 60/40) TT (pause) 0.85 × 0.95 × 0.69 = 0.557 0.