Response to Riley, 16 September 2026
A successionist pushes back in three places, and one of them lands.
@RileyRostovM on X/Twitter skimmed through the sequence and wrote a long comment in response, which I thought would be interesting to respond to! I’ll do this in three parts; first I’ll reprint the tweet verbatim, which I think stands alone well,1 then I’ll go through do a quote-response.2
The tweet in full
Interesting writing and much to consider. My objections to part 2 (and not just from a self-interested POV) are that how do you ensure pausing all frontier AI development (beyond Astra) today does not merely result in an entrenchment of the current order and an establishment of eternal digital serfdom based on who currently controls terrestrial compute capital? To me never transcending our biology and prioritizing staying 100% human ensures that we will never make it past the great filter and we will never escape this rock. Because then machine intelligence becomes a tool to service base human desires, control other human beings, etc rather than the next step in evolution.
But as for part 1, section 2/section 3: My view is that God (through a process of theistic evolution) gave us some physical process within each of our brains something resembling a state or feedback loop that ensures consciousness and a soul. I think any sufficiently advanced computation can replicate this and replicate qualia. Because if God could set up natural laws and processes to create it, then we can discover those natural laws and processes. In the hypothetical, if we can discover what this loop is and how to replicate it, and how to transfer it between substrates with a sort of continuity guaranteed, what would be the moral objection to doing so? That we are somehow morally bound to our flesh containers that only last 85 years and start breaking down after 60? Yeah of course some caution is needed in investigating this and developing ASI in general so we don’t “brick our save game” so to speak and develop a successor with nobody home like you mention. But nevertheless I still think we have a moral obligation to figure it out.
Personally, my view is that we should investigate the substrate/continuity problem ASAP, because while I’m not a hard misanthrope, I am a successionist in that I feel that the current iteration of our species line is so flawed and fallen that solving our issues without augmentation at the minimum is intractable. Our biological limitations and all the cognitive flaws that result are too binding. We simply do not have the hardware (wetware?) to solve the world’s problems as-is. If we could all be walking around with a safe BCI to a sovereign superintelligence cluster of our choosing, that would be a start.
My point-by-point
Let’s start with the philosophical stuff.
I think any sufficiently advanced computation can replicate this and replicate qualia.
(I disagree, naturally, and the reason is that competence is not evidence of it.)
Because if God could set up natural laws and processes to create it, then we can discover those natural laws and processes.
There are a lot of ways to read this claim; given the surrounding context, I’m reading it as a kind of intelligent design “real recognizes real” type statement. That is to say, if an intelligent designer can set up a system, then surely an intelligent observer within the system can derive all of its laws through sufficient empirical observation, even the most mysterious stuff out there. So we should, at minimum, hold out hope for qualia, even if we disagree with the ‘sufficiently advanced computation’ thesis. Right? I don’t think so, for several reasons, and I will list a few of my reasons to be cautious.
One point against comes from the map-territory distinction. The Universe is real big, and I am a real small part of it. So it’s not possible for me to keep a perfectly detailed copy of the Universe in my head at any one time, because otherwise my head would have to be the size of the Universe. Claims that, discoverable natural laws exist everywhere in the observable universe seem overconfident to me out the gate for that reason alone - there could well be regions out there that simply do not behave like our own neck of the woods, and would probably be impossible to instrument the interior of as a result despite having some kind of effect on us. (Now that I think about it, the interior of black holes would be a good candidate of such a region we obsessively theorize about but almost certainly won’t empirically prove at any point.) If we expand our definition of regions to include more conceptual territory, then qualia, to me, seems like it belongs there. A lot of this is just a rephrase of the problems I talk about in part 1.
We have concrete examples that seem to suggest that the Universe really can just withhold certain kinds of secrets from us. The uncertainty principle in quantum mechanics, for example, is pretty well supported, and one extrapolation is simply that you just can’t have perfect knowledge of what a system is really doing to the last detail. You don’t have access to whatever source of RNG God might be using to “generate” reality, so to speak. Indeed, you don’t even know whether God is using a true random number generator or a pseudo random number generator with a really big cycle - they produce identical looking results to someone whose only access is from the inside, watching the random numbers fly by. (I actually don’t consider the uncertainty principle to be as strong an example of this effect in action as I do the existence of, say, asymmetric cryptography for this reason, but it’s commonly known enough to lead with.)3
If one takes seriously the notion that the Universe might itself be a simulation, this secret-withholding capability could have material consequences. We don’t actually have any promise that the simulation is bug-free, after all, and a whole industry’s worth of counterexamples to the notion that any software ever really could be made such. Much like how a speedrunner might perform a certain precise button input, at a certain precise moment, perhaps there exists some physical states that, upon entry, cause natural law itself to go haywire and stop acting like it once did. False vacuum decay might be held up as an example of this theoretical possibility.
But really I like to argue from the reality I observe around me where I can. To me the strongest argument is simply that I interact with laws and processes that I cannot discover even in theory every single day, and those were made by mortal, fallible man, not the zookeeper. The companies I hold investments in have vast troves of processes and information I will never know anything about, nor will even be told about. As a software developer, I both build APIs and work against other peoples’ APIs every day that I will never really be able to discern the immutable laws of. There’s no guarantee that I should be able to regenerate whatever those were after the fact. And yet the world keeps spinning. Things keep happening despite the fact that I understand almost none of it. So it just doesn’t seem like that much of a leap to suppose that maybe someone would create a reality where I can’t understand all of it. Phenomenologically speaking, there’s no real difference between the two.
In the hypothetical, if we can discover what this loop is and how to replicate it, and how to transfer it between substrates with a sort of continuity guaranteed, what would be the moral objection to doing so?
My primary objection is epistemological, not directly moral. I simply don’t think the loop can be discovered or replicated, to be clear. You’re taking a leap of faith every time you step in the Transporter.4 Much like the stock market, past success is no guarantee of future success. It’s always possible things go wrong and you p-zombify yourself, and from there we are in serious “here be dragons” territory re/ getting you back.
I see two obvious ways to cash that out to an ethical claim. The weaker one is that killing yourself is usually bad, and you should usually try to avoid doing it. Life is great! You’re why any of this matters at all, after all. But, death is inevitable, it happens to almost all of us before we reach even a mere hundred years old, and so if you are nearing the end of your life anyway but just don’t feel like it’s your time to go, you might as well take the leap of faith. Who knows? You might get lucky.
The more serious moral claim comes from the implications of this:
he current iteration of our species line is so flawed and fallen that solving our issues without augmentation at the minimum is intractable. Our biological limitations and all the cognitive flaws that result are too binding. We simply do not have the hardware (wetware?) to solve the world’s problems as-is.
I would contest the wetware question as just obviously false, looking at the history of human progress. However, even if I didn’t, walk through the maximally damning chain here for giggles. You claim, maximally, that there is no way to generate radical life extension technology without ASI. I claim, maximally, that developing ASI carries with it the near certainty of human extinction. Taking all of these together, the question is no longer “Why am I personally being told not to live forever?” but “Why am I personally being told not to gamble with the present and future lives of the next most likely sentient beings to myself in the cosmos, on the remote chance that we (or maybe even just I myself) might be allowed to exist for slightly longer?”
some caution is needed in investigating this and developing ASI in general so we don’t “brick our save game” so to speak
(I just like the phrase “brick our save game” to describe my view, great analogy.)
ow do you ensure pausing all frontier AI development (beyond Astra) today does not merely result in an entrenchment of the current order and an establishment of eternal digital serfdom based on who currently controls terrestrial compute capital?
The first thing to do is to observe that the world, as it exists today, does not seem to exist in anything close to those conditions. There’s no digital serfdom, and no one party has anything remotely close to a monopoly on terrestrial compute. I see no signs that “capitalism as usual” would trend towards such a playing field, and every sign that it levels it. Competitive markets compete away all of the profits over the long run and approach perfect competition.
Establishing a entirely new regime very different to our existing one is of course a much less likely question than “won’t this simply entrench the current order”, of course. Mostly I think this is just a misunderstanding of how the mechanism would actually work in practice, but as a contrarian I always like to ask: Why would that be so bad?
Last I checked people weren’t throwing themselves off of rooftops during the 75 years or so that Bell Labs held its own monopoly over the entire telecoms industry and telephone system in the United States.5 That seems like it would have been much more costly to the lives, of much poorer people than today, than if OpenAI and Anthropic somehow merged and established a century long monopoly over Astrolabe, the combined Astra-Fable suite. Moreover, the US today actually has a lot of experience in dealing with natural monopolies, and the usual prescription is to use boring old rate-of-return regulation: Your power company gets to earn, say, 3% profit over cost on every joule of electricity sold, up to a certain cap where it starts to dip down.6 Consumers pay a little more - but electricity is so universally useful it’s worth paying for anyway. It just doesn’t seem like that big of a deal to me. Tangentially, it’s finally worth noting there are a lot of investors who really, really like businesses that generate extremely reliable returns over long periods of time - pension funds, for example, or the hedge funds which cater to them. You don’t actually need to beat the S&P 500 to make your investors happy if your stock lets them hedge against downswings well enough, etc.
That being said, there’s an international component to this, and that’s actually why I do not think we would end up in a long-term permanent duopoly, even if that doesn’t really faze me considering what we’re facing. Fine insured bounties are really good at deterring people from pushing the frontier forward. But it’s not crazy to imagine some country drawing the line at Astra or Fable and saying, “It’s okay to train new models up to this level, but no farther.” And the instant there is an Astra- or Fable-class open model available, there goes the duopoly in practice.
As a broader point I think this is one of those cases where you really need to sit down and actually think through the exact mechanisms by which extinction bounties, or fine-insured bounties in general, work, and not try to analogize them to “just another regulatory capture mechanism”. They genuinely do not work like that. An FIB really does generate a decentralized panopticon, because everyone in the world suddenly has the possibility to make a lot of money if they catch someone transgressing the law. I sometimes liken it to throwing a boulder in a river. Much like how the water doesn’t argue, it just reroutes around the boulder as best it can, so too would capital writ large simply reroute itself to the next most productive investments it can make without running headfirst into the need to pay huge statutory fines.
Like, what are you going to do? Go in front of the judge and say “Your Honor, it’s true my client was training models an order of magnitude above our largest ones, but they had a license from the American Computing Association.” Fine insured bounties don’t care about your little ’license.’ “Fellas, if we all just keep quiet about this I bet we could make something even bigger than Astra.” A conspiracy is only as strong as its loosest-lipped member, and every additional dollar one could make by turning in one’s co-conspirators acts as a wonderful social lubricant.7 All the usual methods of regulatory capture just … don’t work. It’s brilliant in its elegance.
Astra is GPT-6 Astra, the model the pitch names as the ceiling a statute would draw the line just above; Fable is Claude Fable 5.1. They are the two frontier releases of 2026, and between them they are what “the frontier” currently means. I use both names throughout as shorthand for a capability level, not as a claim about either lab. ↩︎
Riley argues under his own name, in public, and I am arguing with a view he chose to publish. That is the whole of it: see non-targeting. If he would rather be cited here under a handle instead, he can have that for the asking. ↩︎
Worth being precise about what the cryptographic version rests on. That one-way functions exist at all is a conjecture, not a theorem: proving it would prove P is not equal to NP, which nobody has done. What we have is decades of concerted failure to invert the particular functions we use, which is exactly the kind of evidence I am appealing to. The point does not need the theorem. It needs only that a world can be arranged so that running a process forward is cheap and running it backwards is not, and we have built such a world out of parts we understand completely. ↩︎
The canonical philosophical version is Derek Parfit’s teletransporter, at the opening of Part III of Reasons and Persons (1984). Parfit’s use of it is the opposite of mine: he thinks the case shows that personal identity is not what matters, and that the copy on Mars should be as good as survival. My objection is not to his argument but upstream of it. He is asking what would be lost if the copy is conscious. I am asking how anyone would find out whether it is, and I think the honest answer is that nobody could: the upload that worked and the upload that did not will file the same report. ↩︎
Dating it from the 1913 Kingsbury Commitment, which bought off the federal antitrust case and blessed the Bell System as a regulated monopoly, to the divestiture that took effect on 1 January 1984, the run is 71 years. It is not a clean example in my favour, either: the 1956 consent decree forced AT&T to license its entire patent portfolio royalty-free, which is part of why the transistor spread as fast as it did. A monopoly that is made to hand over what it discovers is a very different animal from one that is not, and if I am going to lean on this analogy I should say plainly that the licensing was doing a lot of the work. ↩︎
A simplification, and worth unpicking, because the real thing is better for my argument than the version I gave. Rate-of-return regulation does not set a margin on each unit sold. The regulator sets an allowed return on the rate base, the capital the utility has sunk into serving customers, and rates are then set to recover costs plus that return. Recent allowed returns on equity in US electricity have clustered somewhere around 9 to 10 percent, which sounds far more generous than my 3 percent until you notice it is a return on capital rather than on revenue. The governing standard is still the “end result” test from FPC v. Hope Natural Gas (1944): a rate is lawful if the return is enough to attract capital and maintain the utility’s credit, and the regulator does not have to justify the method it used to get there. The relevant fact for this argument is that a century of American law already knows how to let a monopoly exist and still bound what it extracts. ↩︎
This is the objection I take most seriously of the ones aimed at the mechanism itself, and the arithmetic behind the breezy version above is worked out in page 12. The short of it: the payoff to defecting has to exceed each conspirator’s share of the upside from staying quiet, and that condition gets harder to satisfy as the conspiracy gets smaller and richer. A four-person lab with a trillion dollars at stake is a genuinely harder case than a four-hundred-person one. ↩︎