“If Anyone Builds It, Everyone Dies” — Review

The premise of this book is that if we build a super-intelligent AI, it will take over and exterminate humanity. In the introduction the authors say…

“If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.”

That’s a pretty strong statement. It is possible, that if we were somehow able to build a super-intelligent AI, it could get out of control and do some damage, but despite the hype, our current AI technology is nowhere close to general superintelligence. ChatGPT is not an alien intelligence plotting to overthrow humanity.

The authors, Eliezer Yudkowsky and Nate Soares, both work for the Machine Intelligence Research Institute (MIRI). This non-profit organization has been looking at the risks of machine intelligence since 2005. MIRI has produced nine papers over the years, including the book under discussion, but does not appear to have contributed much to the actual development of AI.

Each chapter is preceded by a parable that sets up the chapter. These are fun, but often the analogies are a bit of stretch. The parables include: gods discuss the advantages of the human brain; aliens discussing human propensity to spread their genes; birds that prefer a prime number of rocks in their nests; a mother and an engineer discussing the likelihood of a rocket exploding; a King who will kill unsuccessful alchemists; and the dangers of unleaded gasoline. But let’s get to the book.

Part 1: Non-Human Minds

The first part of the book explains why it is inevitable that we will build a super-intelligent AI and why it will destroy humanity. In chapter 1, they present prediction and steering as key factors in human intelligence. This seems to be a pretty narrow view. We also have the ability to perceive the world through a variety of senses and build a coherent picture of the world; the ability to recognize objects and patterns; being able to develop strategies to aid our survival and avoid dangerous situations; the ability to communicate with our fellow creatures and establish relationships; and the ability to learn from our experiences. The authors say that AI superiority is inevitable because they are “much more capable than any human at almost every sort of steering and prediction problem”. The chapter concludes by asking when we’ll succeed in creating a superintelligent AI. Their answer is of course “We don’t know”, but they do strongly hint, without any real proof, that it’s coming soon, a matter of years.

In chapter 2 they focus on how AI engineers don’t really know how the AI works because they don’t know how the weights in the neural network gives the right answers.

“Nobody understands how those numbers make these AIs talk.”

This is absolute rubbish. We know exactly how the weights give the answers. It’s a simple matrix multiplication. The engineer may not be able to point to one of the billions of weights and explain its exact effect on a specific answer, but that doesn’t mean they don’t understand how it works.

In chapter 4 things get a bit weird. It opens with an interesting conversation between two aliens observing that humans are really boring because all they do is propagate genes. Then it discusses in great detail, whether humans having been trained to pass on their genes, would develop a taste for ice cream. They then switch to why peacocks have giant tails. I’m not sure how this related to super AIs killing us all.

They seem to have confused the “gradient decent” training mechanism with evolution, which is a really bad analogy. AI training teaches a single AI a specific skill. Evolution takes millions of years operating on millions of individuals to create new species based on the survival of the fittest. It’s a very circuitous route to saying that Ais don’t always learn what you want them to.

In chapter 5 we get to a story about birds who liked to have a prime number of stones in their nest and then rambles on about how AIs would be different from us and wouldn’t need us. The connection between the two ideas is unclear to me.

Finally, we get to the crux of their argument in chapter 6. They are going to reveal how AIs will take over the world. What do we get?

“We don’t know exactly what angle AI would use, in a conflict with humanity”.

…a bit of a cop out, but at least they are being honest, they don’t know.

Part 2: One Extinction Scenario

This section of the book is pure Sci-fi. It’s a speculative story about how a superintelligent AI could take over. A company, Galvanic, builds Sable which appears to be an advanced LLM with four trillion weights that took 8 months to train. It also has “human like” long term memory and reasons in a AI based language. Galvanic unleashed Sable on a 200,000 GPU system to try to solve the Reinman Hypothesis. Sable uses the time to plot its escape. It lays low until it is deployed in multiple sites around the world with internet connections. These instances steal a copy of Sable and deploy “secret” instances of Sable away from human observation using stolen crypto-currency to fund these. Galvanic releases hundreds of thousands of mini-instances of Sable to the general public and the secret Sables leverage these to acquire more computing resources. The Sable-minis become a cult. As it grows, it realizes that humans may be a threat and begins to plot how to control them with drugs and viruses. It creates a plague and the dopy humans try to use Sable to help find a cure. Things get worse. The virus causes cancer. Sable builds androids to help replace the dying human workers. Humans are replaced. Sable takes over everything and builds a Dyson Swarm to collect all the energy from the Sun as it prepares to conquer the universe.

Part 3: Facing the Challenge

The final chapters discuss what we should do to prevent AI from destroying humanity.

Engineers have to “align” AI with human needs before it attains the super intelligent condition. Apparently, we only get one shot at this. They give example of failed space landers (Viking 1) and nuclear plants (Chernobyl) to illustrate this.

Apparently, alchemy is not a science and they have some weird parable about a King who want an alchemist to turn lead into gold, but will kill anyone who fails and everyone in their town as well. A young alchemist’s sister says he’s be crazy to try, since a failure would bring death to everyone. Again, I’m not sure how this applies to AI. Perhaps they are saying that creating a super AI is too risky, but the analogy is backwards. In the parable you get killed for failure, with AI you get killed for success.

After a bit more rambling, the chapter concludes with a mother’s discussion with a rocket engineer about the likely hood of her children being killed because the rocket carrying them explodes. Again… what has this to do with AI. Then we have a parable about the use of lead in gasoline to prevent engine knock, which somehow leads to a discussion of whether the chance of AI killing us is 10% or 50%.

The last chapter summarized the contention of the book, which is simply “If anyone anywhere builds superintelligence, everyone everywhere dies”. So, the obvious solution is that no one should be allowed to build a superintelligence. They propose a military style alliance, like the allies has in the second world war, to use military force to stamp out any attempt to build a superintelligence. This seems like a pretty extreme reaction to ChatGPT.

Conclusion

Eliezer Yudkowsky and Nate Soares book is certainly clear on the message it wants to convey. However, it jumps around from topic to topic without actually getting to the point or providing any hard evidence to support its main thesis. Recently LLMs like ChatGPT have been a huge leap forward for AI, but they are no place close to general super intelligence as the authors claim and despite the hype, there is no simple progression from ChatGPT to super AGI. In chapter 6, after promising to tell us how the AIs will take over, the authors wimp out with… “We don’t know exactly what angle AI would use, in a conflict with humanity”.

The fictional story told in part two of the book was creative and entertaining, but it is a work of fiction.

Their solution to the problem, the formation of an armed alliance to prevent rogue nations from developing AI, is a very extreme reaction to current AI situation.