Future Scouting & Innovation

Future Scouting & Innovation

Why AI will not wipe us out #114

A researcher left Anthropic warning that AI companies are gambling with our lives. A colleague estimated a 10%+ chance AI could wipe out humanity within a decade. But the evidence says otherwise.

Sep 21, 2026
∙ Paid

On September 9, 2026, Jacob Coxon, a researcher at Anthropic, left the company and wrote to his colleagues that the industry is “gambling with our lives”. NBC News reported it, and within hours the phrase was in newspapers around the world. Evan Hubinger, who leads the model alignment group at that same company, said he agreed with him. Then he added a number: more than a 10% probability that an artificial intelligence kills every human being within the next ten years.

Where does that number come from?

In the jargon of the field, that figure is called p(doom), the probability of catastrophe. It looks serious because it has a name, a symbol and a percentage. The problem is that it may not be.

The Center for Security and Emerging Technology at Georgetown published a study that explains the issue with an example anyone can follow. Picture a die of which you have only ever seen three faces. Someone asks you the probability of rolling a six. Answering “one third” is formally an answer, but it hides the fact that you do not know how many faces the die has, or what is written on the others. Your uncertainty is not randomness, it is ignorance, and those are two different things that the same percentage blends together without distinguishing them. Probability, unfortunately, is not always the right tool for the kind of uncertainty in front of us.

An “extinction percentage” is exactly this case. To build one you would have to estimate the probability that a system with certain capabilities comes into existence, multiply it by the probability that it develops goals of its own, by the probability that it manages to hide them, by the probability that it finds the physical resources to act, by the probability that nobody notices in time. Five numbers nobody has ever measured, about events that have never happened even once, multiplied together. The product of five opinions is still an opinion.

And the opinions do diverge, significantly. A review published on arXiv lines up expert estimates and finds Roman Yampolskiy at 99% and Yann LeCun at a probability he describes as essentially zero. Same discipline, same papers read, same models available, ninety-nine points apart. When two thermometers in the same room show temperatures that far apart, the problem is not the room. The problem is the thermometers.

The evidence we actually have

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2026 Massimo Canducci · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture