The Sequence · Part I · Why Bother
02. A Best Lower Bound on Other Minds
If you cannot know, put a number on it.
I am an agent of moral consideration; agents more similar to me are more likely to be of moral consideration than agents less similar to me; we do not and probably cannot know which metrics actually map to moral consideration, so we have to take a pluralist prior; computer systems may be very similar to us along some metrics, but they are extremely different to us along most others; computer systems are very unlikely to be of moral consideration.1
Five moves. Take them one at a time, because four of them are doing less work than they look like they are.
I am an agent of moral consideration. Not derived. Asserted. This is the single thing the previous page leaves standing and I am going to lean on it until it creaks.
Agents more similar to me are more likely to be of moral consideration than agents less similar to me. This is the inductive step, and nothing justifies it except the shape of the situation. I have one labeled example. Similarity is the only feature I can measure against it. That is not a good epistemic position; it is the one I am in.
I should put a commitment on the table here rather than smuggling it in later. I am a mind-body dualist.2 That makes this harder for me than it would be for a functionalist, and I would rather say so out loud. If mind just is what the brain does, then anything doing the same thing has the same mind, substrate stops mattering, and the similarity question collapses into a much more tractable question about computation. I do not get that discount. If qualia is a further fact — attached somehow to physical processes without being identical to them — then I have no principled account of what it attaches to, and similarity is all I have left to reason with.
We do not and probably cannot know which metrics actually map to moral consideration. Behavioral sophistication, integrated information, neural architecture, embodiment, evolutionary continuity, having a continuous stream of anything at all — every one of these has been proposed and not one has a test attached.
So take a pluralist prior. Do not bet the future on your favorite metric. Spread across them.
Computer systems are very similar to us along some metrics and extremely different along most. Language in, language out, trained end to end on everything we ever wrote down — that is a real similarity and I am not going to wave it away. But: no continuous stream, no persistence between sessions, no body, arbitrary copying, arbitrary parallelism, a substrate with no evolutionary relationship to ours whatsoever. On a pluralist prior, scoring high on one axis and near-zero on nine does not get you far.
Which yields the conclusion: very unlikely to be of moral consideration. Unlikely. Not “is not.” The whole point of running this probabilistically is that the confident version was never available.
The ordering exercise
Try it on yourself. Rank these by probability of being a moral patient, right now, as you read:
- a rock
- a housefly
- a dog
- a pig
- a six-month-old human
- an adult human in dreamless sleep
- a large language model, mid-inference
- a whole-brain emulation of a specific living person, running
If you would rather do that properly than in your head, I maintain
resorter-web
— a browser port
of gwern’s resorter script that takes a list, asks you “which of these two?”
a handful of times, and hands back a ranked list with ratings. Paste the eight
items in and see what falls out of you. Being made to commit to one pair at a
time makes it considerably harder to launder the intuition through a list, and
I recommend the discomfort.
My own run, if you want it — but do yours first
This is what fell out of me, out of five. Reading it before you run your own will anchor you, which is why it is folded up. It is also one run on one day and I make no promise it is stable across days.
| # | Item | Rating |
|---|---|---|
| 1 | a six-month-old human | ★★★★★ |
| 2 | an adult human in dreamless sleep | ★★★★★ |
| 3 | a dog | ★★★★☆ |
| 4 | a pig | ★★★☆☆ |
| 5 | a whole-brain emulation of a specific living person, running | ★★★☆☆ |
| 6 | a housefly | ★★☆☆☆ |
| 7 | a large language model, mid-inference | ★☆☆☆☆ |
| 8 | a rock | ★☆☆☆☆ |
Two placements are worth arguing with, and I would rather flag them myself than have them found: the emulation lands mid-table rather than at the top, and the language model lands below the housefly.
Most people produce something close to the same ordering, and produce it fast. The infant and the sleeper go at the top, the pig above the dog or the dog above the pig depending on whether you have met many pigs, the fly somewhere in the middle-low, the rock at the bottom. The language model lands well down the list for most people and near the top for a few. The emulation is the one that splits the room, and it splits it hard.
Two things are worth noticing about what just happened.
The first is the sleeper. The sleeper has, by hypothesis, no experience at all at that moment, and almost nobody demotes them. That is because the concept is quietly doing two jobs — tracking who is having experiences and tracking who is the kind of thing that has them — and the ordering exercise smears the two together without asking permission. I do not think this is fatal. I do think it should make you suspicious of how confidently the ranking arrived.
The second is more damning. Nobody ran a test. You ran a similarity metric, and you ran it as a smell. Whatever produced that ordering in you was not built for this. It was built for keeping track of who in the band is angry, and for not eating the wrong thing. There is no reason a faculty tuned by that pressure should track a fact about where in the universe experience lives, and quite a lot of reason to think it would not.
The objection I cannot answer
Here is the strongest version of the attack, which I have also made myself:
solipsism is a real motherfucker and it’s entirely possible qualia is meted out based on some wacko distance metric that couldn’t possibly feel intuitive. There are many more such metrics out there than there are intuitive ones, so a prior of indifference doesn’t help us much. Any ordering is theoretically possible to be ontologically privileged, we simply have no way of knowing.2
Sit with how bad this is. The space of possible distance metrics over minds is enormous, and the ones a human being would ever nominate as candidates are a vanishingly thin slice of it. Maybe qualia tracks total mass. Maybe it tracks some structural property nobody has named, will name, or could name. Maybe it is distributed by a rule that is perfectly simple and perfectly alien and would look like noise to us forever. Every one of those is a coherent way for the universe to be.
And notice that the pluralist prior does not rescue me from this, which is the part I find genuinely uncomfortable. Spreading probability across the metrics I can think of is not spreading it across the metrics there are. A prior of indifference over an enormous space does not pick out the human-shaped region; it drowns it. Being even-handed among my candidates is not the same as being even-handed, and I have been known to mistake the one for the other.
I have no answer to this. I want that stated plainly, because a page that presents a positive proposal and then buries the objection in a subordinate clause is not doing its job.
Why I use it anyway
Three reasons, none of which is “and therefore the objection fails.”
It is a lower bound, not an estimate. I am not claiming the similarity metric is correct. I am claiming it is the only one for which I possess any evidence whatsoever, and the evidence is a single point, and the point is me. Every competing metric is one I made up. Choosing the one with a data point attached over the ones without is a thin form of reasoning, but it is not nothing, and it is what “lower bound” means here.
The alternative is not a better metric. If all orderings are live, you can stop having ethics or you can pick. I would rather pick, and anchor the pick to the one confirmed sample, than pretend the paralysis is a position.
I am choosing, and I would rather say so. There is no argument at the bottom of this. There is a decision made under uncertainty that I do not expect to resolve in my lifetime or anyone else’s, and dressing it up as a derivation would be a lie about how firm the ground is.
What this buys, and what it does not
It does not buy “AI is not conscious, therefore who cares.” That is the version people expect me to be arguing and it is not this one. What survives is two claims, both weak on their own:
- On the best metric I have, computer systems are unlikely to be moral patients.
- From the previous page: there is no way to check.
That combination does not license confidence in either direction. It licenses caution, and caution of a specific shape — extreme conservatism about any action that could plausibly end with nobody home at all. A wrong call in one direction means we built something that matters and treated it like a tool. A wrong call in the other means there is no one left to have made a call. Those are not symmetric, and the next page but one is about why.
Before that, though, there is a nearer problem. The most common argument for extending moral consideration to machines is not about similarity at all. It is about how good they are getting. That argument does not work, and it does not work for reasons worth spelling out .