Why OpenAI’s Astra Could Make AI Doom Harder to Prevent

TLDR Provide a non-technical introduction to opaque reasoning. Present evidence for why opaque reasoning is bad. Discuss the major pitfalls of this approach. Gleaming hope amongst the chaos. Excerpt from AI 2027 The City That Slowly Lost the Ability to See the Council's Decisions For centuries, a city made every important decision in a public hall via a council. Whenever the council chose to build a bridge, go to war, raise taxes, or imprison someone, the members of the council were to debate aloud. The arguments made were often messy, and people sometimes lied. Sometimes the council even gave bad reasons for good decisions, or good reasons for bad ones. But time and time, through debate and consensus, people always came to some sort of understanding. Over time, the demands of the city became larger. A rival city was expanding massively; this led to a feeling of unease since the city had never encountered such a rival. The city's problems started rising too. Trade agreements involved thousands of pages. Wars depended on information that could not safely be discussed in public. Financial crises required decisions within hours. So the council proposed the creation of small private committees to address the growing needs. There is nothing wrong with this since the rival city incorporated a similar approach. This would ensure that the rival city would not catch up to us and the private committees could optimise for the safety of the city as an important goal! At first, these committees were allowed to deliberate privately for only a limited period, roughly four times longer than an ordinary public discussion, before returning to the council with their reasoning. The private committees were also only allowed to discuss sensitive details, after which they had to go to the public hall and explain how they reached their conclusions. This worked well, as the private meetings made decisions faster, while the public explanations preserved accountability. Then the committees were allowed to keep a little more of their discussion confidential with time. There were good reasons for this too. Some arguments involved military intelligence. Some revealed weaknesses in negotiations. Some would cause panic if misunderstood. The city remained fairly functional, so the boundary started moving a little more. Soon, the council no longer heard the full discussion. It received only a small summary. But in return, the gains were massive: The city started seeing exponential progress. Trade flourished. Military decisions were taken quickly enhancing safety. Less money went into this approach, as compared to hosting a fully fledged council. The summaries were also very clear. “After considering the available evidence, we recommend raising the tax.” “After reviewing the security situation, we recommend closing the eastern gate.” “After examining the negotiations, we recommend rejecting the treaty.” The decisions were usually good. And because they were usually good, nobody had much reason to demand the thousands of pages of deliberation behind them. Years passed. The committees became more specialised, and their internal procedures became more complicated. Infact the committees started documenting and dicussing in a private language that allowed them to communicate more effeciently internally. Even members of the council would have struggled to follow the reasoning when trying to go through the documents. Eventually, almost all important decisions were made before they reached the public hall. Then one year, a committee recommended something extraordinary. It wanted emergency powers with no clear end date. The council asked why. The committee gave a careful explanation. There was a foreign threat. The economy was fragile. Several institutions were failing at once. Everything it said sounded plausible. Some council members were suspicious. They asked to see the deliberations. But there was no longer a useful record. Much of the discussion had happened informally, across private meetings with reports that are illegible. Even the committee itself could not reconstruct exactly how the decision had formed. The council could only inspect the final recommendation but not all the thinking that went into this decision. It could no longer inspect the process that had produced them. The council was outraged, they barged into the private commitee's chamber and started acquiring all the documents, they belived the members of the commitee were planning something fishy! They started decoding the documents to see if there were discussions inside the documents that warranted unintended behaviour. Midst this turmoil, the private committee ordered the arrest of the council. The public demanded to know why. A perfectly reasonable explanation arrived that afternoon. By then, the gates were already locked and treasury from most of the city had been wiped. Turns out, all of the money was invested soley into building massive walls sorrounding the ci