Comprehensive FAQ on AI risks

Since AI safety has become incredibly popular recently, I quickly finished the draft I’d been working on for a long time and am finally publishing a comprehensive FAQ on AI risks. It’s also available on my website in a more polished version, where it will be updated regularly. It’s intended for people who are new to the field after seeing all this news and posts on X, and who don’t understand why AI would even want to kill us. That said, people who are already familiar with the topic might also find a couple of interesting points. I was in a real hurry to finish this text and wanted to make it understandable to as wide an audience as possible, so I apologize in advance if you find any serious mistakes. Please let me know about them, and I’ll correct them. Although I wrote the text by hand (without AI), I used AI to translate it into English (since it’s not my native language and I was in a hurry while the topic is still trending among newcomers). Please let me know if you find any parts that were translated incorrectly. AS SHORT AS POSSIBLE: AI continues to develop, and in the future it may be capable of killing every living thing on this planet. It does not need consciousness or humanoid robots to do this, as shown in films. Our typical ideas about ‘evil’ AI prevent us from seeing the real sources of the problem. Misalignment with human values is the main danger, and researchers believe that solving this problem is incredibly difficult. Progress in safety is not keeping pace with progress in AI capabilities. We need to take serious measures right now, as approving and implementing them could take years. Not convinced? Read the objections ↓ PART 1. ‘I HAVE JUST LEARNED ABOUT AI RISKS’. 1. It is not intelligence at all. You can call it whatever you like if you dislike the word ‘intelligence’. You can call it a system that can generalize, plan, write code, use tools, find information, operate interfaces, and carry out tasks. It just takes too long to say or write all of that. It is much easier to use the word ‘intelligence’. 1.1. It just predicts the next word![1] The key word in this sentence is ‘just’. It downplays an actual capability by explaining its underlying mechanism. This is rather like saying, ‘The brain just transmits electrochemical signals.’ But that does not make the brain’s capabilities disappear. It is also important not to underestimate prediction. It is an extremely powerful capability. AI does not just predict the next word; it predicts it correctly. It is also worth thinking about what intelligence actually is. Perhaps prediction is precisely the defining feature of intelligence in general. There is an entire theory of how the human brain works which argues that it is largely engaged in trying to predict the future, then checking its predictions against reality and adjusting subsequent attempts based on the information it receives. 1.2. AI does not understand what it is talking about. We can argue endlessly about the word ‘understands’. What is understanding, anyway? I suspect that AI does understand something, in the broadest sense of the word: advanced models perform very well on benchmarks (evaluation systems) filled with many previously unpublished academic questions and logic problems. For example,  Mythos was able to use amino acid sequences it was given to accurately predict whether a viral capsid (shell) would fold correctly. In other words, it has some idea of how this should happen, even in such a complex and highly specialized subject. But people often use understanding to mean having a conscious conception of something. There is a well-known argument called the Chinese Room, which, in short, says that a system that appears intelligent may not ‘understand’ anything internally or possess consciousness. This has nothing to do with AI risks, since almost all AI-risk scenarios do not assume that AI is conscious. In fact, the design of the transformer, the most common AI architecture, is not all that complicated at its core. Much of it is based on knowledge usually taught in school linear algebra lessons. However, even if you break the system down into its components and study its smallest elements, that does not make it stupid or weak. Yes, you can now proudly declare, ‘It only multiplies matrices!’ But understanding this does not make the danger of AI or its remarkable capabilities disappear. 2. It has no consciousness / soul, so it is not intelligent / dangerous. The soul is not a scientific concept at all: we have no signs or evidence of its existence. Consciousness does exist, but it is not required for AI to be dangerous. Consciousness is a word that most people understand differently. But most philosophers would agree that it is not related to intelligence or reason, but rather describes the capacity to have experiences. Whether AI has experiences is a question that does not affect its capabilities or its ability to harm humanity. [2] 2.1. It has no desires, int