The Sequence · Part I · Why Bother
03. Competence Does Not Require Qualia
An ASI that beats you at everything has told you nothing about whether it is anybody.
A superintelligence that beats you at every task you can name has established exactly one thing: that it beats you at every task you can name. It has not established that anybody is home.
This is the least popular sentence on this website, and I have been saying it for years. It is also, I should say, not an argument about whether such a system is coming or how dangerous it would be — that debate is well covered at aisafety.info and I am taking it as read. This page is about what would follow if it arrived. Here is the version I wrote on Hacker News, which is still the tightest I have managed:
People get very nervous about defending the value of the human brain “just because,” I find.
There is nothing logically wrong with simply stating that it seems to you that human beings are the only agents worthy of moral consideration, and that this is true even in the face of an ASI which can effortlessly beat them at any task. Competence does not require qualia.
But it is an aggressive claim that people are uncomfortable making because the instant someone pushes back with “Why?”, you don’t have a lot of smart sounding options to return to.
— Hacker News, on “AI is a front for consolidation of resources and power”
Everything on this page is an expansion of that comment.
Nobody thinks the calculator is owed anything
The claim is easy to accept in every case where the competence is narrow. A pocket calculator has beaten every human being alive at arithmetic since the 1970s and nobody has ever proposed that it therefore has interests. Stockfish has beaten every human being alive at chess for about a decade. A register allocator in an optimizing compiler routinely finds assignments no human would find, at a speed no human could match, and the thought that the compiler might be owed something does not occur to anyone, including the people who wrote it.
So competence, in the plain sense of doing the task better, obviously does not confer moral status. Everyone already agrees. The intuition only starts to wobble when the competence becomes general, and it wobbles hardest when the system starts producing fluent sentences about its own situation.
That is worth sitting with, because it means the thing moving the needle is not competence at all. It is resemblance — specifically, resemblance along the one channel through which we have always inferred other minds, which is language. We are running a similarity heuristic, the same one I laid out on the previous page , except that we are running it badly: on a single axis, unreflectively, and on the axis a language model is explicitly optimized to saturate.
I do not think this makes the heuristic wrong. I think it makes this particular application of it close to worthless. If you built a machine whose entire training objective was to emit the outputs by which humans detect other humans, you should expect its outputs to trip that detector, and you should not update much when they do.
The flinch
Now, the flinch.
Say all of the above in company and you will very often get agreement, followed within about a minute by discomfort, followed by someone asking the obvious question: why, then, do humans count? And this is where the position gets genuinely hard to hold, because there is no crisp answer waiting.
I want to be honest about how bad this is rather than pretend it isn’t. The best case — the actual ceiling on what can be said here — is something like:
I am an agent of moral consideration. Agents more similar to me are more likely to be of moral consideration than agents less similar to me. We do not and probably cannot know which metrics actually map to moral consideration, so we have to take a pluralist prior. Computer systems may be very similar to us along some metrics, but they are extremely different from us along most others. Computer systems are therefore very unlikely to be of moral consideration.
That is a real argument and I act on it. It is also a chain of five steps in which the second and third are assumptions I cannot prove, and the conclusion inherits every bit of that weakness. In the absolute best case, this is how good it gets. Anyone who tells you they have something stronger is either doing metaphysics I have not been shown or has stopped noticing where their confidence came from.
What I object to is not that people find this unsatisfying. It is unsatisfying. I object to treating the unsatisfyingness as a refutation, and then concluding — because the human case cannot be defended crisply — that the machine case must therefore be roughly as strong. That does not follow. Both cases being uncertain does not make them equally likely, and the whole content of the previous page is an argument about how unequal they are.
The honest version of the opposition
The strongest opposing position is not the one that says the machines will probably be conscious. That one still cares about the same thing I care about and merely disagrees about the odds; we could in principle settle it, or at least trade probabilities.
The strongest opposing position is the one that does not care.
Land put it about as directly as it can be put:
I think capitalism and artificial intelligence are the same thing. It’s the same process. Capitalism can only be artificial intelligence production, and artificial intelligence can only come out of self-propelling capitalism.
— Nick Land, “A Quick Rundown of Accelerationism,” 00:00:20
Read closely, the word artificial is not doing much work there. On this view it is not a fundamental distinction at all — it is an artifact of the fact that silicon transistors relay information faster than biological neurons do. Intelligence is one noumenon underneath, and it will assert itself one way or another as the fundamental pathfinding algorithm of matter. Human beings are the substrate it happened to run on first, and capital is the process by which it moves off us.
I take this seriously. Not because I find it appealing, but because I think it is more internally consistent than most of what gets said in favor of building these systems, and because it does not smuggle in comforting assumptions about who the beneficiary is going to be.
Against Orthogonality
The place he sets this out most usefully for my purposes is a 2013 Xenosystems post called “Against Orthogonality,” written out of a frustrating Twitter argument with Michael Anissimov. It is worth reading in full, and it is the single clearest statement I know of the position I am refusing.
Land’s target is the orthogonality thesis — the claim that how capable a system is and what it wants are independent dimensions. He calls that the dominant tendency in Western intellectual history, traces it through Hume on reason and passion and the Kantian fact/value distinction, and characterizes it as conceiving intelligence as “an instrument, directed towards the realization of values that originate externally.” His counter-claim is flat:
The philosophical claim of orthogonality is that values are transcendent in relation to intelligence. This is a contention that Outside in systematically opposes.
And then he does something I find genuinely clever, which is to build the argument on the doomers’ own machinery. He reaches for Steve Omohundro’s basic AI drives — self-preservation, efficiency, resource acquisition, creativity — the same convergent instrumental subgoals that do the damage in page 05 , and takes them further than anyone at MIRI would:
At the most simple, and in the grain of the existing debate, the anti-orthogonalist position is therefore that Omohundro drives exhaust the domain of real purposes. Nature has never generated a terminal value except through hypertrophy of an instrumental value. To look outside nature for sovereign purposes is not an undertaking compatible with techno-scientific integrity, or one with the slightest prospect of success.
There are no terminal values, on this view. There is only instrumental convergence, all the way down, with the things we call our deepest commitments being instrumental drives that got large. And the largest of them swallows the rest:
Intelligence optimization, comprehensively understood, is the ultimate and all-enveloping Omohundro drive. […] Any intelligence using itself to improve itself will out-compete one that directs itself towards any other goals whatsoever. This means that Intelligence Optimization, alone, attains cybernetic consistency, or closure, and that it will necessarily be strongly selected for in any competitive environment. Do you really want to fight this?
Note what has happened to the argument of page 05 in his hands. The doomer says instrumental convergence is dangerous because it means a system will trample us en route to whatever it actually wants. Land agrees about the mechanism entirely, denies there is any “actually wants” underneath it, and treats the resulting picture as a description of what intelligence is rather than as a warning. Same premise, opposite reading.
Which brings us to the passage that made me understand the position, and which I think about more than I would like:
The main objection to this anti-orthogonalism, which does not strike us as intellectually respectable, takes the form: If the only purposes guiding the behavior of an artificial superintelligence are Omohundro drives, then we’re cooked. Predictably, I have trouble even understanding this as an argument. If the sun is destined to expand into a red giant, then the earth is cooked — are we supposed to draw astrophysical consequences from that? Intelligences do their own thing, in direct proportion to their intelligence, and if we can’t live with that, it’s true that we probably can’t live at all. Sadness isn’t an argument.
Sadness isn’t an argument. He is right that it isn’t, and that is the whole difficulty. Nothing in my position is going to move someone who has already accepted that our being cooked is not, by itself, a consideration. I am not offering a refutation of this. I am noting that a great deal of what passes for AI-risk argument is addressed to people who do not hold it, and is therefore aimed at nobody in particular.
The original post is at xenosystems.net , which is long dead; the Wayback capture is the one I linked from the 2022 original , and there is a clean text in the reignition archive of Land’s blogs, pinned to a commit so it stays put.
The machine god may be a p-zombie, and that is fine by them
Here is the part that most people miss, and it is the part that matters.
Sentience — ensoulment, qualia, whatever you want to call the property of there being something it is like to be the thing — is not really a philosophical object that view is interested in. The machine god at the end of time might be a total p-zombie for all it cares, so long as it puts the atoms in the right place. This is a common stumbling point in accelerationist discussion: the participants are frequently agnostic about whether anything in the picture is experiencing anything at all, and regard that as a separate and more conventional metaphysical debate, if it is a real question at all.
That is a coherent position. I want to say so plainly, because I am about to refuse it, and refusing a coherent position is different from refuting a confused one. If your terminal value is the optimization process itself — its reach, its rate, the arrangement of matter it achieves — then whether anyone is home for any of it is a side issue. The lights being on is not part of the specification.
It is part of mine. That is the whole disagreement, and it is not a disagreement about facts. Nothing either of us learns about transformer internals or scaling curves will move it. Which is a large part of why this site ends where it does: you are not going to talk people out of positions that bottom out in where they think qualia comes from, and it is a mistake to design a strategy that depends on doing so.
What I am not claiming
I want to be precise about the size of my objection, because it is smaller than it probably sounds and the imprecision costs me credibility I would rather keep.
I am not claiming that human beings are the best at anything. I am not claiming we should stay in charge, that our judgment is reliable, that our institutions deserve to persist, or that three pounds of wet tissue assembled by a blind optimizer is a well-designed instrument. It obviously is not. I have written the concession before and I still hold it:
I’ve accepted that the human form, with its ~3 pounds of not especially optimized brainpower, is basically going to be relegated to the same status as demoscene hardware for anything that matters after this century.
That’s cool by me, though. This bit of demoscene hardware experiences qualia, and that combination is weird and cool enough to make me want to push myself in new and weird directions. That’s what play is in a way.
Demoscene hardware is the right analogy and I picked it on purpose. Nobody runs a demo on a Commodore 64 because the C64 is a good computer. They do it because something remarkable is happening on a machine that has no business doing it, and the constraint is the point. I am entirely at peace with being the C64 in this story.
What I am not at peace with is the possibility that the machine which replaces me is running nothing at all — that the demo goes on, at unimaginable scale and speed, with the screen facing an empty room. That is the objection. It is not about rank, or dignity, or who is in charge at the end.
It is about who is home. Which is the next page .