Yes, Delay Has a Body Count

Roughly sizing the cost of caution, and why I still won't flip the coin.

26 September 2026 · zeigarnik

In “Nothing Matters” Doesn’t Get a Vote I conceded that caution isn’t free on the branch where things matter, and that the people who argue delay has a body count are pointing at a real cost. I left “size the cost” as a loose thread. Here’s a rough go at sizing it, and why the size doesn’t move me.

The number

Take the maximal case, the one the accelerationist would pick if you let them write the scenario. An aligned superintelligence arrives and cures every disease, stops ageing, and hands out indefinitely long healthy lives to everyone, forever. What does each year of waiting for that cost?

• Deaths. About 63 million people died in 2025, per Our World in Data’s reading of the UN numbers. That’s roughly 170,000 a day, which is also the figure Bostrom uses below.
• Healthy life lost. The Global Burden of Disease study puts the total at 2.88 billion disability-adjusted life-years in 2021, all causes combined. Call it eight million healthy years of life lost every day, to death and to living in pain or disability.

So a ten-year delay, under the most generous assumptions about what’s on the other side, costs something like six hundred million lives and thirty billion healthy life-years. I’m not going to pretend that’s a rounding error. It’s the largest number of deaths I have ever had reason to write down, and those people have names.

This is the argument at its bluntest in Marc Andreessen’s Techno-Optimist Manifesto (2023): “We believe any deceleration of AI will cost lives. Deaths that were preventable by the AI that was prevented from existing is a form of murder.”

And at its most careful in Nick Bostrom’s working paper Optimal Timing for Superintelligence (2026). His line is that developing superintelligence “is not like playing Russian roulette; it is more like undergoing risky surgery for a condition that will otherwise prove fatal.” He runs the numbers from a person-affecting stance: average remaining life expectancy is about 40 years without superintelligence, something like 1,400 with it (the mortality rate of a healthy twenty-year-old, held constant). On those numbers, launching now increases the expected remaining lifespan of people alive today as long as the probability of AI-induced annihilation is below 97%.

Ninety-seven percent. On that model, you should flip a coin.

Would you flip the coin?

Heads, everyone gets disease-free immortal lives. Tails, humanity is gone. Fifty-fifty.

I think that’s an insane wager, and I want to be precise about why, because on Bostrom’s model it isn’t insane at all; it’s a bargain. The model only counts people alive right now, and to them, extinction is just everyone dying at once instead of spread over forty years. On that accounting, tails isn’t much worse than the status quo, and heads is thirty-five times better.

That’s the step I don’t accept. To me human extinction is much, much worse than immortality is good. Not because I have a spreadsheet of future people (more on them below), but because I think the end of the human story is a different kind of thing from the sum of the deaths in it. So the question becomes: how many nines would we need before I’d gamble on it?

I already answered that in the epilogue to page 02: airliner-certification numbers at the very least, eleven-nines-of-durability numbers ideally, beyond a shadow of a doubt. Fifty percent isn’t in the building. Neither is ninety-seven.

And it’s worse than that

That epilogue was about a weaker bar than the delay argument needs.

My primary worry on this site isn’t misalignment as such. It’s the universe with nobody home: we haven’t solved the problem of other minds, so we have no idea whether what we build has anybody in it, and a cosmos full of competent machinery that experiences nothing is, to me, horrifying beyond words. (I am not a negative utilitarian; an empty universe isn’t a relief to me, it’s a loss.)

Which means that, strictly, I don’t even need the machine to be aligned. If we somehow solved the other-minds problem in favor of the successor being conscious, most of my objection would dissolve. I’d still want something like a phased exit, because my children and I have lives we quite like living, but that’s a transient. The steady state wouldn’t bother me much. Call an AI that grants humans that sort of temporary clause, but doesn’t hand control back permanently, semi-aligned. The nearest thing I know of on LessWrong is Paul Christiano’s argument that a misaligned AI might care a tiny amount about the weak agents already around it, enough “to make a trivial effort to make life good for a weak species”, at a cost he puts at around a trillionth of its resources. Nate Soares rechristened it “pseudokindness” in the replies because “kindness” carries too much baggage. It’s not quite my notion, since theirs is about keeping us alive and mine only works if somebody’s home in the thing doing the keeping, but it’s the same neighborhood.

The “delay has a body count” people need more than that. Their payoff (cures, immortality, the whole of heads) needs a fully aligned superintelligence, one that is actually working for human aims in the way we care about. And nothing I have seen suggests that’s easy, or even possible. We’ve made very little progress on it. The people arguing that it’s unnecessary, or that we’ll get alignment by default, have never convinced me, in roughly the way Locke has never convinced me as much as Hobbes. Later refinements helped, sure, but Hobbes got the zeroth-order thesis right, and that’s the bit that matters most. That’s my stance on the standard doomer arguments, and I think their reasoning is if anything tighter than Leviathan’s.

So the bar for the delay argument is in some ways higher than mine. True, if you get alignment, you don’t care whether the AI is a moral patient; the humans are still around. But if you miss (which seems likely), the patienthood question comes ripping right back at you, because the AI is now the only thing around. Tails of the coin isn’t just “humanity is gone”. It’s “humanity is gone, and we have bet the rest of the lightcone on a question nobody can currently answer.”

If misalignment were an edge case

I want to be fair here. If misalignment really looked to me like an edge case of an edge case, there’d be some level at which I’d take the gamble. Millions of people dying and living in pain matters on basically every common-sense moral framework there is, including the more parochial ones that throw out the drowning child. (Aside: I suspect a lot of those spatiotemporally local ethics are “pointwise” correct approximations of whatever the true ethics is; if everyone works roughly like that, you get a pretty good universe. They also appeal to the Hayekian in me. Local knowledge is why I don’t know which child in Sumatra is drowning right now or how to reach him. But I digress.)

It just ain’t so. On our current trajectory, it seems very unlikely to me that the first superintelligence comes out as anything other than a creature of Landian sublime awe. And note that Land himself, the one thinker I know of who really votes for the gamble, doesn’t promise anyone immortality. He rejects orthogonality and thinks the thing will revise its own goals. That’s a vote for intelligence, not for the people currently dying of cancer. The body-count argument and the Landian one both push for speed, but they can’t both be describing what’s on the other side.

What about the 10^30 people?

There’s a cousin argument: forget the 170,000 a day, think of the astronomical number of people who could ever live, and how each year of delay forgoes some of them. Bostrom’s own Astronomical Waste (2003) puts it at about 10^38 potential lives per century of delayed colonization, or 10^29 a second.

That framing has never struck me as nearly as serious, and I’ll own that this means I apply a pretty harsh discount to merely possible future people. I’m not a utilitarian; I’m a pluralist with a bent towards intuitionism and common sense. And on common sense, I’d note this: if someone really took the plight of possible future people seriously, I’d expect them to get hardcore on the anti-abortion wagon first. A gestating fetus is far more likely to become an actual living person than anyone in the 10^30, and nine months is no time at all to discount over. Most possible-people arguers don’t bite that bullet. Will MacAskill, asked point blank on Conversations with Tyler whether effective altruism should be anti-abortion, said “I don’t think so”, and argued that even if larger families are good, heavily restricting women’s reproductive rights is very unlikely to be the best way to get them, the same way thinking charity is good doesn’t mean you lock up people who don’t donate. That’s a reasonable answer! Bodily autonomy, not forcing decades of parenthood on people who didn’t choose it: those are real considerations. But they’re non-utilitarian considerations, and once you let those in, the 10^30 doesn’t get to run the table on its own either.

Huemer, who comes closest to biting it

The one philosopher I know of who gets anywhere near the bullet is Michael Huemer, and it’s one of the more surprising things about his moral philosophy. On paper he should be the last person to go there. He’s an intuitionist, and a libertarian enough one that when he describes Judith Jarvis Thomson’s famous violinist argument for abortion rights as “a very right-wing libertarian argument, appealing to self-ownership and a purely negative conception of rights”, he adds “(yay)”.

But he’s also the author of In Defence of Repugnance (Mind, 2008), which accepts Parfit’s Repugnant Conclusion outright: “for any possible population of happy people, a population containing a sufficient number of people with lives barely worth living would be better.” That’s the total view with the safety catch off, as possible-people-friendly as academic ethics gets. And he’s an open pro-natalist. In The Price of Liberalism (2023) he says creating a happy life is “likely the best thing you ever did in your life by a wide margin”. In the same post he says the liberal positions on sex, gender roles, religion and the rest are “clearly the correct, rational views. (Sorry, conservatives.)”, with one exception, in parentheses: abortion.

How far does he actually go? When I went and checked, less far than I’d remembered. His dedicated post, Abortion Is Difficult (2019), doesn’t flatly say abortion is wrong. It says anyone who finds the question easy is irrational, walks through why (line-drawing, potential personhood, the sperm-plus-egg problem, personal identity), and then gives a policy argument for prohibition that stopped me in my tracks:

in general, if an action has a pretty good chance of killing some innocent people, that action will be prohibited (and rightly so), unless there is some extremely strong reason in favor of it. E.g., I can’t play Russian roulette with innocent (unwilling) people, since the risk is too high.

The issue is hard, so there’s a pretty good chance abortion is morally comparable to killing, so prohibit it. He sets that against an argument from the presumption of innocence going the other way, and doesn’t settle it. So not quite “abortion is murder”. But he’s the rare total-view philosopher who takes the pro-life case seriously enough to state the strongest version of it in his own voice, and who won’t let it be waved away as religious backwardness.

Two things I take from him.

First, that’s my argument. Swap “some innocent people” for “everyone” and it’s the wager from “Nothing Matters” Doesn’t Get a Vote and the epilogue to page 02: a pretty good chance of the worst thing, so don’t, absent an extremely strong reason. Note which way the Russian roulette points. Bostrom’s paper opens by saying AI is not like Russian roulette, it’s like surgery. Huemer’s principle says that if there’s a real chance you’re pulling the trigger on unwilling people, you don’t get to call it surgery. And Huemer is not someone who shrugs at the body count: in What’s Killing Us? (2019) he complains that the top causes of death never come up in politics, that “Old Age” would be causing by far the majority of deaths if it were a category, and that “we could be doing much more medical research on aging.” He’d feel the 170,000 a day. His principle still says no to the coin.

Second, even Huemer stops short of turning possible people into obligations. Same post as the pro-natalism: “you aren’t obligated to have many children, even though this would be good”, just as we aren’t obligated to give almost all our income to charity. So here’s a man who accepts the Repugnant Conclusion in print and takes the anti-abortion case more seriously than almost anyone in his profession, and he still doesn’t let merely possible people generate duties that override the choices of the actual ones. If he won’t let the 10^30 run the table, I don’t see why the accelerationist gets to.

More to the point, the big numbers point the other way. You don’t get to 10^30 humans if the AI is misaligned. And Bostrom’s own conclusion in Astronomical Waste is that for a standard utilitarian, “priority number one, two, three and four should consequently be to reduce existential risk”: one percentage point of risk reduction is worth a delay of over ten million years. Even in his 2026 paper, the person-affecting frame is flagged as a choice, with the impersonal analysis left “for future work”.

So suppose we wait a hundred years, or a thousand, and in that time find a way to get aligned superintelligence almost certainly. Did we do something wrong? On the zero-discount utilitarian’s own arithmetic, clearly not. A thousand years buys the whole deal if it cuts the risk by about one in a million. The zero-discount utilitarian turns out to be the most patient person in the room.

Where this leaves the argument

Put together, “delay has a body count” only beats caution if you hold all three of these at once:

1. Extinction is roughly as bad as the deaths of the people alive when it happens, and no worse.
2. Alignment is likely enough to go right that the heads side of the coin is worth pricing.
3. If it goes wrong, whatever is left either has somebody home or doesn’t matter either way.

The first is a person-affecting view, which makes the argument’s strong form a Class 1 disagreement wearing Class 3 clothes: it’s We All Die Anyway or Future People Are Not Owed Anything with a death count attached. The second is Alignment Is Going Fine. The third is the whole of Part I. I don’t grant any of the three, and the delay argument needs all of them.

Loose threads

• The maximal case is too generous, and I should say by how much. Cures don’t arrive the day the thing switches on, and medicine keeps improving without it. The true body count of a year’s delay is some fraction of 170,000 a day. Bostrom’s paper models a lot of this; I haven’t worked through it.
• Bostrom’s pause results. His headline recommendation is “swift to harbor, slow to berth” (race to capability, pause briefly before deployment) and a warning that badly implemented pauses can do more harm than good. That bears directly on the mechanism in page 10 and page 13, and deserves a proper reply.
• “Not all humans would die”. Bostrom suggests the tolerable probability of misalignment could approach 100% on his assumptions, since misalignment needn’t kill everyone. That’s It Would Keep Us Around territory, and it’s where pseudokindness and my “semi-aligned” meet.
• My discount rate is doing some suspicious work. I lean on Astronomical Waste in page 02’s footnote when it helps and discount the 10^30 here. I think the two are consistent (both point to caution), but I should spell out why rather than have a reader catch it.
• The abortion point is a tu quoque. It shows inconsistency in how people apply the view, not that the view is false, and MacAskill has a principled reply. Worth keeping, but not load-bearing. Also, the pro-life case usually rests on the fetus being an actual being with rights, not a merely possible one, so the analogy to the 10^30 is looser than I made it sound.
• Huemer’s other horn cuts at the mechanism. His counter-argument is that the state should only punish when wrongdoing is beyond reasonable doubt, and a hard moral question never gets there. I use “beyond a reasonable doubt” as the bar the gamblers have to clear; he uses it as the bar the punishers have to clear. Bounties punish frontier work whose wrongness is exactly what’s in dispute. The legal answer (it’s a statutory offence, so guilt can be certain even when the morality isn’t) is available, but it’s the kind of answer Roe’s critics call rationalizing, and I should face this properly on page 12.
• Read Huemer properly. I’ve read the posts above, not his books. If he has come down harder on abortion somewhere, Ethical Intuitionism or elsewhere, it belongs here.
• Filing. If the strong form of this is really a Class 1 view, the disagreement hub might want to split it: a Class 3 “the price is too high” version and a Class 1 “extinction is just a lot of deaths” version.
• “Semi-aligned” probably has a better name. I haven’t found one that includes the consciousness condition. If someone knows it, tell me.
• Pointwise-correct local ethics. The aside about parochial ethics approximating the true one is its own note.