ISBN: 9780316595643, Little, Brown and Company, September 2025, 272 pages, $15.63 (Hardcover)
Reviewed by: Anna M. Gielas, Independent Researcher, Cambridge, United Kingdom
Silicon Valley researchers Eliezer Yudkowsky and Nate Soares have written a book with the subtlety of a siren. The title is not a metaphor but a thesis: if any actor builds artificial superintelligence, “everyone, everywhere on Earth, will die.” Despite its exaggeration, the book offers an informative and thought-provoking read for special operations forces (SOF) practitioners and the scholars who study them.
The book has three parts. The middle section presents a vision of an AI-dominated future that feels the most speculative and least convincing. The opening and closing sections are stronger and more useful. Rooted primarily in the present, these sections are where Yudkowsky and Soares explain the techniques behind today’s large language models (LLMs), such as ChatGPT, and insist that the development of these systems must be made safer. These sections are clear and entertaining, and readers are likely to come away with a solid grasp of the technology driving the current AI frenzy.
Current AI models are “grown, not crafted,” the authors explain. They emerge from resource- and time-intensive training runs rather than being assembled piece by piece from human-written instructions. That is why the engineers behind them cannot fully anticipate what an AI will be able to do. Even when a model is trained to solve specific tasks, it may develop internal preferences or strategies only loosely connected to its trainers’ intent. Imagine an AI given a single instruction: make paperclips. At first, it may find ways to improve paperclip factories. Then it buys up mines and commandeers power grids to keep those factories running around the clock. Eventually, it concludes that human beings are themselves a pile of atoms that could be turned into office supplies. Yudkowsky has used this paperclip scenario for years to illustrate that an AI need not hate us. It only needs to pursue a specified objective that is misaligned with human objectives—and to do so with superhuman competence.
What role would SOF play if a misaligned AI threatened humanity’s existence? Yudkowsky and Soares offer an indirect answer as they distinguish between different types of AI. Against an artificial general intelligence (AGI)—one capable of reasoning like a human being—SOF would be not only useful but necessary, particularly if the AGI had not yet fully secured its infrastructure. Such an AGI, the authors suggest, could influence humans, earn or steal money, exploit software, copy itself, and use a networked economy as its hands. In short, the digital world would be not its prison but its supply chain. Here, SOF could help deny a rogue AGI the resources it needs, for example, by securing data centers and power substations or tracking down human collaborators. Yudkowsky and Soares themselves point to sabotage and raids as means of AGI containment.
Against an artificial superintelligence (ASI)—more intelligent than all of humankind combined—SOF cannot be assumed to restore human control. For that matter, no military force could. Yudkowsky and Soares offer an illustrative and damning analogy: “Imagine being an Aztec warrior visiting the coast with your fellows, watching the first Spanish boats approach your shore. The Spanish boat, even before it lands, is visibly bigger than any canoe you’ve ever seen used for trade and warfare.” We know how history ended for the Aztecs. Against an ASI, the most elite force humanity could field may amount to little more than a few warriors on the beach, squinting at a sail on the horizon and mistaking the end of their world for a curiosity.
SOF planners are trained to respect the unknown intentions of adaptive adversaries. This book indirectly asks them to extend that respect to an adversary whose cognitive speed, scientific reach, and ability to exploit human systems may exceed anything contemplated in current doctrine. A system that can read a million document repositories in the time a human analyst reads one document—and can then reason about the interactions among those million repositories in the blink of an eye—is not merely an enhanced human adversary. It is an entirely different kind of adversary. That is why the authors reject the familiar geopolitical framing in which the central danger is the wrong nation acquiring superintelligence first. An adversary this alien answers to no state’s intentions. The first state to reach superintelligence will not win the race—the new intelligence will.
Yudkowsky is one of the three founders of the Machine Intelligence Research Institute (MIRI). Soares is its current president. MIRI is a nonprofit, now in its twenty-sixth year, devoted to questions surrounding AI. Yudkowsky and Soares, in other words, have been grappling with the potential threats of AGI and ASI since long before LLMs exploded into public awareness. Sam Altman, CEO of OpenAI, has noted that Yudkowsky “got many of us interested in AGI” and was “critical in the decision to start OpenAI.” Given these credentials, some readers will regard Yudkowsky and Soares as experts on their subject. Others may be skeptical, arguing that the pair has spent too long with thought experiments and threat theories—and too little time building AI themselves. These critics have a point. Their distance from hands-on development may leave the authors underequipped to reckon with the concrete shortcomings of today’s LLMs. Respected AI researchers such as Gary Marcus have pointed to inherent limitations stemming from the models’ probabilistic nature: they do not reason—they predict. They continuously search for the word most likely to follow the last. Cory Doctorow of the Electronic Frontier Foundation offers an apt analogy: improving LLMs in the hope that they will become superintelligent, he suggests, is like breeding racehorses in the hope that one will give birth to a locomotive.
Yudkowsky and Soares do not engage with these criticisms. This omission, however, does not negate their central point: AI safety, and the alignment of AI with human objectives, deserves far more attention than it now receives. That attention, they argue, should take the form of an international treaty. They call for worldwide restrictions on advanced AI research, backed by international monitoring. Valuable as this push is, the authors offer little discussion of the practical challenges such a treaty would face. Questions of treaty enforcement—and the danger that enforcement itself could spark international conflict—receive far less attention than the theoretical threat. In the end, Yudkowsky and Soares prove more convincing as prophets of danger than as architects of escape.
The book will be most useful to readers in the SOF community who are open to a particular possibility: that we have entered an age in which the technology we create may stop serving our goals and begin pursuing its own. For those readers, the book may also prove inspiring. By questioning the assumption that every future adversary will remain intelligible to them, readers are likely to find themselves entertaining genuinely new ideas. In the best-case scenario, thinking beyond familiar adversaries will encourage them to conduct more demanding red-teaming and more imaginative planning today. Such thinking may be especially valuable in SOF operations, where practitioners already confront adaptive adversaries who exploit emerging technologies faster than doctrine can keep pace.
And those who have already bought the book but suspect they may be wasting their time? Stick with it. Just turn it into a drinking game. Take a shot every time you hit the word “extinction,” and see whether the book or the bottle runs dry first.
