The Sequence · Part I · Why Bother

02. A Best Lower Bound on Other Minds

If you cannot know, bet on it!

Last revised 23 September 2026 · alpha

Another quote from my Hacker News profile, which more or less sums up my thinking even today:

I am an agent of moral consideration; agents more similar to me are more likely to be of moral consideration than agents less similar to me; we do not and probably cannot know which metrics actually map to moral consideration, so we have to take a pluralist prior; computer systems may be very similar to us along some metrics, but they are extremely different to us along most others; computer systems are very unlikely to be of moral consideration.¹

Let’s take these claims one at a time.

I am an agent of moral consideration.

Asserted, not derived. Cogito ergo sum. This is the single thing the previous page leaves standing.

We could cast doubt upon even this if we wanted. Maybe I never really existed in the far past and was only created last Thursday, say. Maybe I exist only in this very instant and neither the past nor the future have any moral weight at all, dude. Far out. But I think all of those hyperliteral solipsisms really fall under the next banner.

P(moral patient | like me) > P(moral patient | unlike me) …

Reality does seem to have some kind of bias towards distance functions.

Take gravity, for instance. If you are one light-year away from me, the gravitational pull I exert on you and you exert on me back is more than if you were two light-years away from me, or five, or ten. The edit distance between the two words cat and cot is less than the distance between cat and fly. And if I want to guess whether I have high odds to get along with a new person or not, I could do better than I would otherwise by asking myself how close or far away they would be from me in the five-dimensional space formed by our Big Five personality scores - except for Neuroticism, maybe, but even that’s not immediately obvious (neither my wife nor I are exactly first percentile low neurotics, but we still clearly like each other an awful lot).

Things which have lower distances between them often seem to be more like one another, and sometimes even have similar properties to one another, than things which are “farther away” in some sense. Virtually everything I notice actually being useful in real life seems to wrap around something distance function-related. So here I take an inductive leap: I posit qualia is too, or at least, the odds that a being experiences qualia is.

It’s not impossible, of course, that qualia is truly randomly distributed. Some region of spacetime here, another region of spacetime there, say. Such a possibility just seems to run counter to how the rest of reality works. I don’t think the ground is quite that soft.

… but we don’t know what kinds of “like” and “unlike me” sway the needle most

solipsism is a real motherfucker and it’s entirely possible qualia is meted out based on some wacko distance metric that couldn’t possibly feel intuitive. There are many more such metrics out there than there are intuitive ones, so a prior of indifference doesn’t help us much. Any ordering is theoretically possible to be ontologically privileged, we simply have no way of knowing.²

Behavioral sophistication. Integrated information. Neural architecture. Embodiment. Evolutionary continuity. Having a continuous stream of anything at all. Every one of these has been proposed as the distance function that determines whether a being experiences qualia in some form, either its presence at all or - more ambitious by far! - “how much” qualia there is.

But as aforementioned, you can’t actually test for that correlation, let alone causation. I see why they might seem intuitively appealing to many monists, but alas, I am a mind-body dualist. I’m so sorry. All of my experience being a certified hood classic moral patient suggests that mental and physical events are just fundamentally different kinds of things.

One might suspect that my solipsism is driven by my dualism, but it’s really the other way around. Solipsism is the truly hard question that makes dualism seem more intuitive to me. Let’s take the monist options one at a time; if you are a functionalist, then the mind just is what the brain does, then anything doing the same thing has the same mind. Substrate stops mattering. If you’re an identity theorist, then mind is what the brain is, and so substrate matters enormously. Either way, you’ve pinned consciousness to something physical, and then we should be able to envisage a test for mind the same way we can envisage a test for all things physical. But we know how well that went last time we tried.

If, however, qualia is a further fact, attached somehow to physical processes without being identical to them, then there is no hard reason it should be testable in the same way. This looseness is fine, because I also have no determined account of what it attaches to or why. (Not all dualists think this way, to be clear; Chalmers is a dualist, but he happens to think that psychophysical laws probably do track functional organization, which leaves him much more open than me on the question of machine consciousness.) Similarity is all I have left to reason with. So let’s try to reason with it.

… which suggests taking a pluralist prior

Let’s keep hounding similarity. What does this suggest? Don’t bet the future on your favorite pet metric. Spread your bets across them. You’re allowed to VT and chill on this. You probably end up with broadly more defensible moral results and intuitions that way.

Now that’s in some sense an unsatisfying claim. Am I really saying, for example, that you should just take your ten favorite ethicists and act according to an average of their views? Yes, but I think it might be less unsatisfying to think about some counterexamples. Are there cases where people did things that are widely considered extremely not cool by today’s standards, which are in some way attributable to “overindexing” on one or another ground level ethical theory? Or even some notion or other of which like-versus-unlike metrics actually matter for moral patienthood? Definitely!

• René Descartes indexed heavily on the ability for moral patients to put their thoughts into words, and use formal reasoning. Non-human animals were then non-conscious automata, which, seems to me as a former cat and dog owner and current peanut feeder to the local neighborhood crows to be at least somewhat objectionable by reasonable people. This non-conscious conclusion led directly to live animal dissections under the belief that their squeals of pain were no more morally meaningful than the grinding of wheat on a millstone.
• If you really want to give yourself nightmares for the future, look up how doctors routinely operated on newborn infants without anaesthesia until the late 1980s, rooted in a deep medical consensus that newborns were incapable of feeling or remembering pain. It seems to me sitting here in 2026 as a former and current newborn caretaker that I would go absolutely fucking feral on a surgeon who cut my baby without administering anaesthetic.
• Insert your favorite twentieth century dictator here, and there is almost certainly some kind of state-backed ethnic or, more rarely, ideological or class-based genocide ascribed to them. To say nothing of what state leaders did before the twentieth century, even though technology was not advanced enough at the time for them to do quite as much damage on quite such a scale.

Generally speaking, a good person - to the extent we can in our smoky haze discern that - doesn’t go to any of these extremes. That’s both intuitive and backed by the “moral VT and chill” strategy I’m proposing here.

(If you look at this from afar, I am asserting that genuine moral progress has been made as society has advanced. Even with all my probabilistic hedges I think on a person to person basis we are doing far better than we did a hundred or even ten years ago on average. I’m sorry if you thought the guy who thinks you yourself might not actually be conscious was going to use this as some kind of edgelord defense to not care about you, but (a) if you thought this I don’t think you have adequately thought through your mental model of someone obsessed enough with solipsism and its discontents to write something like this, and (b) ethicists do not tend to behave significantly more ethically than the general public anyway.)

An ordering exercise

Let’s play a game to break up all this reading. Rank these by probability of being a moral patient, right now, as it occurs to you intuitively.

• a rock
• a housefly
• a dog
• a pig
• a six-month-old human
• an adult human in dreamless sleep
• a large language model, mid-inference
• a whole-brain emulation of a specific living person, running

I recommend you use resorter-web for this. It’s a browser port of Gwern Branwen’s resorter script I maintain that takes a list, asks you “which of these two?” a handful of times, and hands back a ranked list with ratings. Paste the eight items in and see what falls out of you. Being made to commit to one pair at a time makes it considerably harder to launder the intuition through a list, and I recommend the discomfort.

Most people produce something close to my ordering. What’s more, they tend to produce it fast. The infant and the sleeper tend to go at the top. The pig above the dog or the dog above the pig depending on whether you have met many pigs. The fly somewhere in the middle-low, the rock at the bottom. The language model lands well down the list for most people and near the top for a few. The emulation is the one that splits the room most in San Francisco, but I still feel my placement is roughly the most popular one.

Does this tell us anything? Trivially it tells us people’s intuitions don’t necessarily agree. But neither are they totally uncorrelated, either, quite the opposite in fact. There are over 40,000 ways to permute an 8-element list, but in practice people only pick a few dozen. Even trolls who want to (ugh) rustle your jimmies tend not to pick totally randomly, unless of course that’s the bit. (When intuitions are stable enough across people that you can joke around by subverting them, I feel like that’s deceptively good evidence that they are actually pointing to something real.)

Do these moral intuitions matter at all? We don’t need to relitigate realism vs. antirealism vs. unsure-so-hedge-our-bets-ism, so I’ll just point interested readers to Huemer’s Compassionate Phenomenal Conservatism for a more detailed discussion of this. I think the best answer is, “Sure, why not, if anything would, this would.” In the absence of defeaters, the way something seems to be intuitively counts as prima facie evidence that that thing is actually how it seems to be, and that arguments against that are self-defeating³.

Computer systems are very different to us along most metrics.

Back to the point. In modern AI ethics, people who try to take a more ‘rigorous’ approach to the problem of where qualia come from often risk a kind of ethical overindexing on substrate alone, in both directions. Carbon chauvinists criticize the idea that artificial intelligences can exhibit moral patienthood because they had the bad luck to be formed out of silicon instead of carbon even though silicon is very similar to carbon in all but atomic weight and might even be the second most likely basis for organic life elsewhere in the universe. This seems somewhat overconfident to me, although it doesn’t really cash out to anything I consider obviously ethically unintuitive.

But on the opposite end of things you have people who we might call “complexity chauvinists” - people who overindex on things like… Well, okay, that one gets real complicated real fast. I’ll just describe the pattern I see with the simplest case, that of behavioral mimicry. A “behavioral mimicry” chauvinist would grant equivalent or near-equivalent moral status to a chatbot merely because it walks like a furless primate and talks like a furless primate. Except, actually since it can do this millions of times a second it might actually deserve millions of times the moral status of one such primate, then billions, then trillions… This seems very overconfident to me, and it does cash out to things I consider obviously ethically unintuitive.

There’s also a kind of “negative versus positive definitions” asymmetry here which concerns me. The typical carbon chauvinist posits a negative definition. if you do not have X, you cannot experience qualia - but if you do have X, there isn’t necessarily much being put forward for whether you are “more” of a moral patient than other things with X. You could totally be a carbon chauvinist and think insects matter more than people, for example, these aren’t at all mutually contradictory. However, usually when I see complexity chauvinists put their case forward, there is some kind of implicit power scaling snuck in there. The machine god at the end of time they hope for, for whatever reason, tends to end up as a utility monster - maybe that’s the fate of all new gods …

Unless we build it wrong, of course, in which case, well. Quoth the article:

When I spoke about these ideas with the philosopher Thomas Metzinger of Johannes Gutenberg University of Mainz in Germany, he raised a related point I hadn’t considered. What if we instead accidentally created conscious digital minds that are incredibly efficient at experiencing suffering? What if we created an explosion of negative well-being among synthetic beings, of a depth never before seen in history?

But then that’s just a utility monster with a sign flip, so to speak, a being which we would presumably have an absolute moral duty to make sure does not come into being. … Right?

All of this paints a picture of computer systems, even these AIs we are so concerned about, as being extremely different to the flesh-and-blood me, who I know is actually experiencing something right now. That’s bad news for the computer system when it comes to who I ascribe potential moral patienthood to.

Hence we arrive at our final stop:

Computer systems are unlikely to be of moral consideration.

Let’s just … sit with how bad this situation is, epistemologically, for a moment. The space of possible distance metrics over minds is enormous. The ones a human being would nominate as candidates are a vanishingly thin slice of it. Maybe qualia tracks total mass. In that world the Sun is more conscious than we are. Whatever. Maybe it tracks some structural property nobody has named, will name, or could name. Maybe it is distributed by a rule that is perfectly simple and perfectly alien and would look like noise to us forever. Every one of those is a coherent way for the universe to be. The only data point any of them have to explain is “Why me?”

And yet I still have to act, so here is what I do with all that noise. Notice that an alien metric, whatever it is, is exactly as likely to favor a rock as a person or a GPU. It drags every estimate down toward the same murky floor, but there may be signal that rises above that floor. Given the existence of my moral intuitions around this topic, it does seem feasible to infer that signal comes from metrics I can actually name.

If I’m being fair about it, I should prefer the simpler ones, because simple rules are the kind the rest of physics seems to run on, or at least be approximable by - gravity is approximable with a simple inverse square law relation, Maxwell’s equations are a little harder but still in the realm of things you can teach an undergraduate (ask me how I know). So I take a broad population survey: behavior, architecture, embodiment, evolutionary continuity, having a continuous stream of anything at all. I score high on all of them, trivially, since I’m the reference point. A pig scores high on most. A language model scores high on one, arguably two, and close to zero on the rest.

So: P(patient | me) > P(patient | pig) > P(patient | language model). That’s not a proof, just the best ordering I know how to build. But it is the ordering, and it isn’t close.

Now I’m not smart enough to figure out much beyond these simple inequalities. Indeed I don’t think there is any way to get more precise, mathematically, from here, without relaxing some of our core assumptions, but that isn’t really my goal here. Most of the time none of this stuff matters, after all. Who cares about the problem of other minds when you’ve got groceries to buy? You can get through life more or less unscarred being only 99% sure other people are conscious, or being only 50% sure, or even being only 1% sure.

But weird things start to happen in people’s heads when they’re only one-in-a-million sure.

You know what I mean. I spoke about it before. Behaviors start to come out that, purified by daylight, no reasonable person would be able to look at in full fidelity and say “Seems harmless.” Oh, no one would ever put together that story except for you, of course; they weren’t there, and if they were they wouldn’t care enough, and if they did care enough they’d be you and that’s a reductio ad absurdum because you clearly didn’t. You start to go sociopath. And no one long survives being fuel for that blackened flame.

Gun to my head, if you were to say “shut up and intuit!”, it feels to me like there’s about a 3% chance of computer systems being of moral consideration. Roughly equal to flipping a coin five times in a row and getting heads every time. I don’t personally spend a lot of time trying to work out that number, because as aforementioned I harbor great doubt as to whether any actual math can be done here beyond some simple inequalities, maybe an impossibility theorem or two. I don’t think it’s impossible by any means. I do think there’s a fair bit working against them. I mean, we didn’t even get into the potential anthropic bias problems here; if I am a “randomly selected qualia-haver”, and machine intelligences dominate the future, say, I should almost certainly end up being one of them, right, and not one of the apes that harbinged them?

Epilogue / shortcut

At the risk of cutting to the chase: Three percent is way too fucking low for me to gamble our slice of the lightcone on. You can’t even certify an airliner with that - any failure that could bring it down has to be certified by the FAA as “extremely improbable”, with a likelihood of about one in a billion per flight hour. Like, come on. Ninety percent would be too fucking low! “Beyond a reasonable doubt”, the legal standard for ordinary criminal charges, would be too low, because this is emphatically not an “ordinary criminal charge”. This is beyond Nuremberg if you fuck it up.

We have businesses advertising eleven nines of durability for the data they protect. If you can get us to that level of certainty on the question “Can the machine god at the end of time truly keep alive the fire of experience?”, plus or minus an order of magnitude or two, then alright, we can entertain the possibility of successionism or whatever.

If that sounds like an impossible task for a mere philosopher… Yeah, dude. Philosophy is, like, hard. But your whole deal is being good at arguments, right? Why am I not convinced then? Why is anyone not convinced by your infallible arguments? Why are people so reluctant to hand over the keys?

Are we all just stupid? Then how do you explain the existence of people who are even smarter than you and yet disagree with you? Or maybe you think you are the smartest person alive, and therefore get to dictate what happens to the rest of us. Even if that were the case - then what of those most basic of liberal moral intuitions, summarized by Article 3 of the Universal Declaration of Human Rights?

Everyone has the right to life, liberty and security of person.

Descend, Icarus, please. Your father and I love you dearly.

The vast majority of reasonable people do not agree with you on this plan of action. If the ROI is truly as high as you think it is, waiting for us to collect the incontrovertible evidence in your favor is a small price to pay to hedge the titanic risk you are proposing we take as a civilization here.⁴

One decent place to start might be against the first page of this sequence. If I’m right about the fiendish difficulty of the problem of other minds, then that evidential bar may never come. Then we wait forever. I can live with that. So can you, literally.