The Sequence · Part I · Why Bother
04. The Universe With Nobody Home
The outcome I actually fear, which is not the one usually described.
Here is the thing I am actually afraid of, stated as plainly as I can manage.
I am afraid that we slip, more or less by accident, into a world where no beings at all experience qualia anymore, and that our universe therefore stops having any moral relevance.
That is it. That is the whole terminal value. Everything else on this site is downstream of it.
It is worth noticing how undramatic the scenario is. There is no war in it. It does not require anybody to be malicious, or even careless in any way they would recognize at the time. The lights stay on. There is, quite possibly, a tremendous amount of activity — construction, computation, expansion, optimization, the whole cosmic-scale industrial project that the more enthusiastic futurists like to sketch. It is just that there is nobody home for any of it. The universe goes on doing things, at enormous scale and at increasing speed, and none of it is happening to anyone.
I am not the first person to stand here and it would be strange to pretend otherwise. Bostrom describes the same outcome in Superintelligence as a society of economic miracles and technological awesomeness with nobody there to benefit — a Disneyland with no children, full of intricate structure, lacking any type of being that is conscious or whose welfare has moral significance. Yudkowsky got there earlier, in Value is Fragile : take an agent with every aspect of human value except the valuation of subjective experience and you are left with a nonsentient optimizer that makes genuine discoveries which are never savored, because there is no one there to do the savoring. A universe with no one to bear witness to it, he says, might as well not be.
The difference is one of position rather than content, and it is worth being concrete about what their lists actually contain, because the contrast is where my own view becomes legible.
Yudkowsky’s list
His thesis is that “value isn’t just complicated, it’s fragile” — that there is more than one dimension of human value where, if that one thing is lost, the future becomes null. He works three removals in detail. Take a mind with almost the whole specification of human value and leave out boredom, and it spends until the end of time replaying a single highly optimized experience, over and over. Leave out the idea that feelings have external referents, and you get a mind that goes around feeling as though it has made important discoveries without making any — “having become its own experience machine.” Leave out the valuation of subjective experience and you get the nonsentient optimizer.
The list is longer in the concluding passage: sentient beings, enjoyable experiences, experiences that are not the same one over and over, learning, discovering, freely choosing. The crucial structural point is that these are co-equal. Lose any single one and, on his account, the future is null.
Bostrom’s list
Bostrom’s is more specific still. In “The Future of Human Evolution” (2004) he argues that competitive dynamics could make the things we value stop being adaptive, and he says plainly what those things are:
Much of human life’s meaning arguably depends on the enjoyment, for its own sake, of humor, love, game-playing, art, sex, dancing, social conversation, philosophy, literature, scientific discovery, food and drink, friendship, parenting, and sport.
He then imagines what he calls an all-work-and-no-fun future, in which fitness is maximized by “non-stop high-intensity drudgery, work of a drab and repetitive nature, aimed at improving the eighth decimal of some economic output measure.” And here is the line that separates us:
Even if the workers selected for in this scenario were conscious, the resulting world would still be radically impoverished in terms of the qualities that give value to life.
Where I get off
For Bostrom, the conscious drudgery world is already a catastrophe. For Yudkowsky, a future that loses only boredom — everything else intact, sentience included — is null, on a par with the empty one.
I do not think either of those is the catastrophe I am tracking. I would take the all-work-and-no-fun world. I would take the single optimized experience replayed until the heat death, which I concede is a genuinely awful thing to find oneself willing to accept. I would take a future with no humor, no parenting, no dancing, no art. In each of those there is somebody home, and the ledger is not zeroed. It may be a terrible score. It is a score.
So it is not that I disagree with their lists. Every item on them is a real loss and I would fight about most of them in any ordinary context. It is that they are running a conjunction — all of these, or nothing — and I am running a single term. Everything else on their lists I am, in the last analysis, prepared to lose.
The cost of that position should be stated rather than hidden: it is much thinner than theirs, which is why it needs so much less to get going, and it is correspondingly bleaker. My win condition is compatible with outcomes almost nobody would describe as good. I am not claiming that the presence of experience makes a world fine. I am claiming it is the difference between a world that can be evaluated at all and one that cannot.
The fears this is not
The concern gets routinely collapsed into four adjacent ones, and I want to pull them apart, because I do not hold any of them in the form they are usually stated.
It is not human extinction as such. If humanity ended tomorrow and the universe were otherwise full of beings who experience things, I would find that sad in a parochial way, the way anyone is sad about the end of their own lineage. But the ledger would not be zeroed. Run it the other way to see how little the human part is doing: if every person alive were replaced overnight, one for one, with a perfect duplicate that behaved identically and experienced nothing, no observation would change and I would count it among the worst things that had ever happened. The species is not what I am tracking.
It is not suffering. I want to be careful here, because a lot of people who worry about AI are working from a suffering-focused view, and my position is not a stronger version of theirs. It is a different one. A universe with nobody home contains exactly zero suffering. It scores perfectly on that metric. I do not think it scores perfectly. I have set out why at more length in a note on negative utilitarianism .
It is not disempowerment, or job loss, or the concentration of wealth and control in very few hands. Those are real, they are worth writing about, and several of them are already happening. They are also, importantly, the kind of thing that can be undone. They belong to a different genre — the genre of problems where you can be wrong for fifty years and then fix it.
It is not “the AI takes over.” A takeover by something that experiences things would be, at worst, a political catastrophe. Possibly a very bad one. It is not what this page is about.
The test that separates my concern from all four: picture the outcome in as much detail as you like, and then ask whether anyone is home. That axis is the only one I am reading.
The thinnest thing I need
I should say what I am not, since it is load-bearing and people tend to assume otherwise.
I am not a suffering utilitarian. I am not, honestly, a utilitarian at all in anything but a very loose sense. I have no felicific calculus. I do not think these quantities add up, and most of the standard objections to aggregative ethics land on me about as hard as they land on anyone.
What I do commit to is this: there is such a thing as moral patienthood. Some entities are the sort of thing that can be wronged; others are not. That is a property, it is real, and its extension is not empty.
That is a much thinner commitment than it looks. It does not require aggregation. It does not require me to say how patienthood is distributed, or to have any way of measuring it — page 02 was explicit that I do not. It requires only that the total absence of the property would be a loss. Nearly every ethical framework I know of needs something at least this strong to get started. I have written more about where this leaves me in a separate note ; it is not a comfortable position, but it is a small one, and small is what I want here.
The escape hatch I would like to take
There is a view on which this entire page is a non-problem, and I would like to believe it.
If panpsychism is true — if experience is a general feature of matter rather than something that happens only in certain arrangements of it — then the lights-out universe is not a possible outcome. There is no configuration of atoms with nobody home, because there is nobody-home nowhere. Most of my concern dissolves on the spot.
I am not convinced beyond a shadow of a doubt by panpsychism, although it would certainly ease my concerns. The combination problem seems to me a real difficulty and not a technicality. But I hold the door open, and I have a standing suspicion in roughly that neighborhood:
My running conspiracy theory is that qualia is actually some kind of jailbreak on physical reality. Enough of the systems I’ve seen in real life are leaky at their edge cases that I wouldn’t be totally surprised if even the ones I consider most fundamental are too.
Now notice the shape of what I just did. I described a position, said I would like it to be true, and then reported that I am not convinced. That sequence should make you suspicious, and it makes me suspicious of myself. Wanting a view to be true is grounds for more scrutiny of my reasoning about it, not less. If I ever notice myself growing more sympathetic to panpsychism in proportion to how bad the alternative looks, that is evidence about me and not about consciousness. I have tried to keep the two accounts separate; the note on what panpsychism would buy me is where I work through it properly.
The successionist bet
The people who are relaxed about all this are not villains, and I do not think it helps anyone to write them as villains.
Broadly, they hold one of two positions. Either qualia comes along for the ride with capability — build something sufficiently competent and you have built somebody — or it does not matter whether it does. Page 03 was about the second position, which at least has the virtue of being explicit about what it is willing to trade. This page is about the first.
Stated as a wager, the first position is: sufficiently capable systems will be experiencing systems.
It might be right. I have no proof against it, and I want to be honest that I never will. But look at where the bet is being placed. Page 01 says there is no way to check, not merely that we have not gotten around to checking. Page 02 says the best instrument available is a similarity-weighted prior, and that the candidate in question resembles the one confirmed case along a handful of axes while differing from it along nearly every other. So the wager is being made on an unverifiable proposition, about an unusually dissimilar candidate, with everything on the table.
That is not evil. It is a calibration error.
And solipsism is genuinely weird once you start applying a probabilistic lens to it, because the uncertainty cuts in both directions and people tend to collect only the half they like. I cannot assign zero to the proposition that a large model experiences something. I also cannot assign anything close to certainty. What I notice, though, is that most people making the bet are not assigning a number at all. They are pattern-matching on behavior — on fluency, on apparent deliberation, on the thing saying it would rather not be turned off — which is precisely the signal page 01 says carries no information about the question.
The asymmetry that does the work
Almost every catastrophe I can name has, in principle, a road back. A civilization can lose a war, a technology, a century. The recovery may take longer than anyone alive would like, but there is someone for whom it would be a recovery.
This one does not have that. You cannot restart qualia in a universe that has none — not because it would be technically difficult, but because there is no one left with a reason to try, and no one for whom the trying would matter.
That asymmetry does most of the work in everything that follows, and I want to be precise about how little else it needs. I do not require a high probability of the outcome. I do not require fast takeoff, or any specific threat model, or even that the similarity prior from page 02 is correctly specified. I require three things:
- that the outcome is possible at all,
- that we have no way to check in advance whether we are producing it, and
- that it admits no recovery.
Given those three, extreme conservatism is not a temperament. It is just the correct posture. It is the same reasoning that stops you running experiments on the only surviving copy of something, and it does not become less correct because the experiment looks promising.
An aside on Huemer and the vote
There is a useful parallel in a place you would not expect to find one.
Michael Huemer wrote the best case I know for doing nothing about politics. “In Praise of Passivity” argues that voters are ignorant, experts are worse at prediction than they believe, social systems are harder to reason about than physical ones, and that most political activism is undertaken for the sake of the activist’s self-image rather than the result. The conclusion is medical: in politics, as in medicine, first do no harm. It is a libertarian argument for keeping your hands off the controls, made by someone who genuinely means it.
And then he broke his own rule to argue that people should vote against Trump, in a post titled “I Don’t Care About the Issues,” on the grounds that the issues were not the point. The point was that a sitting president had made a concerted, credible, illegal attempt not to leave office after losing an election, and that this was the one thing worth voting on regardless of where one stood on tax policy or immigration or anything else.
Look at the shape of that exception, because it is exactly the shape of the argument on this page. Huemer’s default is inaction, justified by our ignorance about what any given intervention will do. What overrides it is not a claim that he suddenly knows which policies are best. It is a claim about a particular category of loss: the peaceful transfer of power is the mechanism by which political errors get corrected, it is historically rare and recent, and losing it is not one bad outcome among others but the loss of the thing that lets you recover from bad outcomes. You do not need to know what the right policy is to know that you should not break the error-correction machinery.
That is the same move I am making, one level down. I am not claiming to know what a good future looks like — page 02 is an admission that I cannot even reliably say who would be in it. I am claiming that experience is the thing that makes any of it assessable, that its total loss admits no recovery, and that this licenses a level of caution I would not accept for ordinary risks. Passivity is the right default in both cases. The exception is not “this outcome is very bad.” The exception is “this outcome removes the ability to notice that it was bad.”
Most arguments for taking AI risk seriously ask considerably more of you than this. They ask you to accept particular claims about capability curves, about optimization pressure, about what happens at some threshold. As it happens I find a good deal of that persuasive, and the next page is about why. But I want it on the record that this page does not depend on any of it.