10% Probability of Extinction. What Do We Get in Return?

Picture ten buttons on a table. Nine of them do nothing. One of them kills you. Press one, and you do not know which is which.

Nobody sane plays that game. There is no upside. You are gambling your life for nothing.

Now change one thing. Someone you love, your daughter, your father, your closest friend, is dying. Press any of the nine safe buttons and they live. Press the one bad button and you both go. Same odds. Same table.

I believe most parents would press. I am a parent myself, so yes, I am biased, and some of you would not press, and I respect that. The odds did not change, but the game did, because now the gamble buys something.

Hold that picture. The people building AI put this exact table in front of all of us, and last week one of them finally said the odds out loud.

The number, said out loud

On September 8, an Anthropic researcher named Jacob Coxon resigned publicly, writing that the top labs “are racing straight to self-improving superintelligence and gambling with our lives,” and that “the people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt.” His post passed a hundred million views in a day. He quit two months before his equity vested. He paid to say it.

The next day, Anthropic’s alignment science lead, Evan Hubinger, replied in public:

“Jacob is correct here; we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”

Let that sink in. Not a critic. Not a doomer on a podcast. The person whose job is keeping the AI aligned, at the company with the best safety reputation, says more than a one in ten chance that it kills every one of us, within ten years.

And he is not an outlier. Dario Amodei, Anthropic’s CEO, has put the chance of things going “really, really badly” at 25 percent. Geoffrey Hinton says 10 to 20. A survey of 2,778 AI researchers landed on a median of 5 percent, with a third or more saying 10 or higher.

One honest math moment, because the number deserves it. To be precise, 10 percent does not mean 10 percent of us die. It means a 10 percent chance that all 8.2 billion of us do. But there is a standard way to weigh a gamble like that, the same way insurers price catastrophe: multiply the odds by the loss. Ten percent of 8.2 billion is 820 million. In other words, taking this bet carries the same weight as signing away 800 million lives, roughly one person in every ten you know. Hold on to that number, because in a minute I will put another number next to it.

This is a pattern, not an incident

If one person quits and warns on the way out, that is a story. When twenty do, that is data. Since 2023, at least twenty researchers and executives are out of OpenAI, Anthropic, and Google, most walked, some resigned in protest, a few were pushed out, all tied to some version of the same alarm, and the money several of them left behind is what makes it credible. Daniel Kokotajlo refused to sign a non-disparagement clause and put roughly two million dollars of vested equity at risk to keep his right to speak. William Saunders told the US Senate that testifying could cost him “equity worth millions.” Coxon walked before his cliff.

The pattern even has a shape. The 2024 wave said “my company is deprioritizing safety.” The early 2026 wave said “the industry is repeating old mistakes.” The latest wave says something darker: the race itself is the problem, and no single lab can stop it alone. “There are no adults in the room,” as one DeepMind researcher put it on his way out.

If you want to know more about what each of them actually said, open this.

The other ledger

While we debate a probability, here is what kills us today, every year, with 100 percent certainty.

Two things jump out of this table, and they are conservative numbers, several of these causes are undercounted. The thing we fear most, war, kills the fewest by two orders of magnitude. And the things AI’s builders promise to solve, disease and hunger, kill more than 30 million people every single year. Run that for the same decade the 10 percent refers to, and the status quo costs over 300 million lives with certainty. Now put the two numbers side by side, as promised. The gamble weighs 800 million, a risk. The status quo costs 300 million from these causes alone, a certainty. Over the decade, doing nothing loses close to 4 percent of humanity to things the builders claim AI can fix. That is the honest shape of the table: a certain loss on one side, a weighted risk on the other, and the question of what the risk actually buys deciding between them.

And here is the twist that the doom debate keeps skipping: the same people telling us about the 10 percent are also promising that AI could erase most of that second table. Cures for cancer. Heart disease understood. Hunger solved. Amodei has written the whole catalog, an essay called Machines of Loving Grace, where he predicts a century of medical progress compressed into a decade. That promise is the prize sitting next to the risk. A gamble has two sides, and so far we have only talked about the losing one.

The two questions nobody is asking

Everyone is arguing about the number. Is it 5 percent, 10, 25. Wrong argument. A probability has no meaning until you know what the gamble buys and who you are trusting to deliver it. Which leaves two questions I do not hear anyone demanding answers to.

Question one: where is the framework that guarantees the benefits reach us, and stops the winnings from concentrating?

So far, no cure has arrived. Not one. And even if the cures are genuinely on the roadmap, nobody has told us how they would reach us. Does it trickle down? Is there a price tag on the cure that our shared risk paid for? Nobody has answered, because nobody is being asked.

And even a stated how is not enough without enforcement. Say a lab announces the cure is five years out and everyone will get it. Five years later the cure is real, priced at a hundred thousand dollars, and 80 percent of the population cannot afford it. Those 80 percent carried the extinction risk and got nothing back. That is the heist again, this time with a pharmacy counter.

And while we wait, here is the scoreboard we can actually observe today: none of the abundance has shown up, only the concentration. No cure shared, no price named, no mechanism written down, and meanwhile the money is not moving either. The wealth AI generates seems to be concentrating, and the gap keeps growing. I am fully for capitalism, but capitalism has gone off track when the larger share of the population loses its fighting chance while a handful of companies approach the power of states. So before any talk of upside, the demand is simple: show us the mechanism, in writing and in law, by which AI’s gains get distributed fairly. Not a blog post about future abundance. A framework with teeth.

Why does this come first? Because trust is earned and proven. A promise is just a promise, nothing more. The companies asking us to accept the risk are the same companies promising the reward, and right now the verifiable evidence, where the money is actually going, points the wrong way.

Question two: once that trust is established and enforced, what do we get in return?

Only after question one has a real answer does this one become a negotiation instead of a sales pitch. Let me draw the line first. If the return is that the cure exists but only the wealthy can afford it, while a few companies become unimaginably rich and powerful and the rest of us carry the extinction risk, then 10 percent is not acceptable. One percent is not acceptable. There is no number low enough, because a gamble where one side holds the risk and the other side holds the winnings is not a deal, it is a heist.

But if the return is universal, cancer gone, heart disease gone, hunger ended, lives extended, then the gamble is at least discussable. Ask a parent sitting in an oncology ward whether they would press the button. Their answer is different from a healthy wealthy person’s answer, or from someone with no kids to lose, and both are legitimate, though wanting the bet is their right, not my verdict, and not a yes. Thirty million certain deaths a year is not a small thing to weigh against a probability. So no, I am not saying the risk is acceptable. I am saying the return decides whether the conversation is even legitimate, and today nobody has named the return.

I should say plainly: I did not invent the pieces of this argument. A Stanford economist, Chad Jones, has modeled how much growth could justify how much extinction risk. Stuart Russell calls the current arrangement private companies playing Russian roulette with every human on earth. The labs themselves, Amodei’s “Machines of Loving Grace” most eloquently, have written the upside catalog. What I do not see anyone doing is putting the two together: making the answer to “what do we get, and who gets it” the test of whether the risk is legitimate at all.

Where that leaves me

By my own test, today’s deal fails. No cure has arrived, nobody has said how one would reach us if it did, the only observable result so far is concentration, and the return is a promise made by the people who profit either way. So no, I am not accepting the 10 percent, and you should not either.

But I am also not joining the pure doom chorus, because the other ledger is real. Sixty-two million people die every year, thirty million of them from things AI might genuinely help solve, and a person watching someone they love die has every right to want that button pressed. And I will be honest about my own stake: I am interested in a longer life span too. Call that chasing immortality if you want. I call it wanting a choice in when I expire, which is a slightly different thing. So let me define the framework I keep demanding, as three tests a passing deal would have to show, in this order. The cure test: the benefits reach people as treatments and prices they can afford, not as press releases, and the roadmap says who gets them and what they cost. The oversight test: independent verification with real authority, so “trust us” is replaced by inspection, the way we already treat aviation and medicine. The profit test: the gains are shared by a mechanism written down in advance, not redistributed by goodwill after the fact. That is a negotiation worth having, and we are not having it.

So the next time a lab leader tells you AI is worth the risk, do not argue about their percentage. Ask them my two questions. When does the cure reach us, at a price we can pay, and when does the wealth get distributed, in a way we can verify? And what, exactly, do we get in return?

Maybe I am wrong about all of this. Call me out. That is what the comments are for.

Sources: Coxon’s resignation and Hubinger’s “>10%” reply, Sept 8-9 2026 (Axios, CBS, TIME, Forbes); Amodei’s 25 percent (Axios AI+DC Summit, Sept 2025); Hinton’s 10 to 20 (CBS); the 2,778-researcher survey (AI Impacts, Jan 2024); Amodei’s abundance catalog (Machines of Loving Grace); departure list compiled from CNN, NPR, CNBC, TechCrunch, NBC, Senate testimony, and the leavers’ own posts; mortality data from WHO, IARC GLOBOCAN 2022, IHME Global Burden of Disease 2021, the UN World Food Programme, UCDP, and the UN World Population Prospects 2024. World population base for percentages: 8.2 billion.