Is the Quiet World Stable?
The line the whole site rests on is that it is easier to shift the Nash equilibrium of working on frontier AI than it is to create safe AI. That sentence gets a great deal of work out of a technical term, and I have been using it loosely. A Nash equilibrium is not a state of affairs you approve of. It is a strategy profile from which no player can improve their position by unilaterally deviating, and whether the world this proposal aims at has that property is a separate question from whether it is a nice place to live.
The chilling-effect note describes the destination concretely: almost no claims filed, premiums quoted and paid and boring, unobtainable quotes at the far end of the risk curve, careers routing around the line without anyone announcing anything. That is the state to be tested. This page tests it, by taking each player in turn and asking what they would do if everyone else kept playing as described.
I should say the conclusion first, because it is not the one I set out to write. Three of the four hold up better than I expected. The fourth does not hold up at all, and it is the insurer.
The researcher: exit stops somewhere short of zero
Page 09 argues the effect is exit rather than conviction: that the rational response to declining a bounty is to leave the building, because your exposure is determined by other people’s choices and rises with every hire. I still think that is right. But the page describes exit as a flow in one direction, and a labour market does not work that way.
If capable people leave frontier capability work, the remaining ones become scarcer, and scarcity raises what they can charge. The premium is a cost of holding the job; the salary is what the job pays. People leave while the second does not cover the first, and stop leaving when it does. The stable outcome is not an empty industry. It is a smaller and much better paid one, sitting at whatever wage clears the risk, an ordinary compensating differential, of the kind that already prices commercial fishing and deep-sea welding.
Page 09’s own framing points at this without following it: the quantity that has to move is a difference between two salaries, not the value of a life. If the frontier salary rises to close that difference, the mechanism has to bite again at the new level, and where it settles depends on how steep the labour supply curve is, which nobody knows.
Is that a failure? Mostly not, and this is the one place where the arithmetic rescues the argument rather than the other way around. Exposure runs on , with headcount in the exponent, so a frontier industry that has been repriced into half the headcount at twice the salary is a materially different risk object from the one that exists now. The mechanism was never trying to make the wage zero. It was trying to make small. Paying more per head is exactly how a firm responds to that constraint, and it is the response the design wants.
What does worry me is who stays. The people for whom a large priced risk is worth a large salary are the risk-tolerant ones, and they are not obviously the population you want concentrated at the frontier. That is adverse selection on personality rather than on information, the insurer cannot price it, and I do not have an answer to it.
The lab: deviation is available and is mostly priced
The lab’s profitable deviations are the familiar ones: reimburse the premium, restructure so no individual is legibly a participant, move the work to a jurisdiction that has not adopted anything, or shrink the team below the size at which the arithmetic has anything to work with.
Reimbursement is answered on page 08: the rate stays individually priced, so a lab that reimburses pays more for the same work and has converted an employee’s problem into a line on its own budget, which is where the proposal wanted it anyway. Jurisdictional flight is page 10’s whole subject, and if page 10 is wrong the proposal is wrong. Team size is the clock and it is running.
So the lab’s deviations are known, they are named as the load-bearing uncertainties elsewhere on the site, and none of them is news. I am satisfied that this player has been checked, which is more than I can say for the next one.
The insurer: the price is built out of the thing it prevents
Here is the failure I did not see until I wrote the players out.
Page 08 makes the premium the load-bearing output. Not the fine, not the prosecution: the continuously updated market estimate of how much your colleagues can be trusted, arriving as a bill before anything has happened to anyone. I believe that. The trouble is what an underwriter builds that estimate from.
Insurance pricing is experience-rated. Rates are anchored, ultimately, on realised claims. And in the steady state this proposal is aiming at, there are almost no realised claims; that is the definition of success. So the quantity the premium is supposed to encode, the probability that somebody reports, has no observable frequency to be estimated from, and the price becomes an unfalsifiable judgement about a counterfactual.
Now put two carriers in that market. Both are quoting a large number for work near the line. Neither has paid a claim in six years. The carrier that shaves its rate wins the book and, for as long as the quiet holds, keeps the underwriting profit, because the loss it was pricing for does not arrive. The one that holds its rate loses the business and learns nothing, because it never sees the losses that would have proved it right. There is no feedback that punishes underpricing until the tail event lands, at which point the deterrent was already gone.
That is a price war with no discipline on it, and it runs in exactly one direction. The deterrent decays not because anyone attacked it but because the market did what markets do to a risk with no recent losses. Every quiet year makes the premium cheaper, and the premium is the entire deterrent.
The general shape is worth naming, because it is not specific to insurance: the mechanism’s output is fed by data that the mechanism’s success destroys. Deterrence suppresses the events; the absence of events makes the estimate soft; the soft estimate is a weaker deterrent; the events return. That is not a fixed point. It is a slow cycle, and this proposal sits on a limb of it.
I do not think this is fatal, but it does mean the design is incomplete as written. What would fix it has to be a price input that does not depend on realised losses: mandatory reporting of near misses so the frequency data exists without the catastrophe, third-party audit as an underwriting condition, capital or reserve floors set by statute rather than by competition, or a loss pool that prices the sector rather than the carrier. Every one of those is a rule imposed on the insurance market from outside, which is uncomfortable for a proposal that keeps advertising how little state machinery it needs. That discomfort is the honest cost, and I would rather carry it than pretend the market prices itself.
The insider: two deviations, and they are not equally answered
The insider’s first deviation is silence, and the site’s answer is that silence is not safe: page 09’s point is that declining the bounty leaves you inside a building where everyone else still holds it. Silence is not a stable strategy for the individual playing it, which is what an equilibrium argument needs.
The second is private settlement, and page 12 works it properly: filing produces a liability that is bounded, adjudicated, insured and final; settling produces one that is unbounded, unadjudicated, uninsured and perpetual, owed to a growing set of counterparties who cannot be held to anything. I am persuaded by that, with the caveat page 12 itself flags: an insurer is the one party sophisticated enough to make quiet settlement work as a portfolio strategy, which is the same player who just failed the previous section.
There is a third deviation that nothing on the site addresses, and I want to put it down even though I think it is weak. In a world where nobody collects, the expected return to being the person who notices falls to nearly nothing, while the costs of noticing (the career, the references, the friendships) do not. This is the standard shape of an inspection game, which characteristically has no equilibrium in pure strategies: full compliance makes monitoring unprofitable, and unprofitable monitoring makes offending pay.
The reason I think it is weak here is that this design does not depend on a funded monitoring industry that can starve. The detectors are colleagues who are already in the room and already being paid to be there. Their marginal cost of noticing is zero, because noticing is not the job; it happens as a byproduct of the job. What the quiet does erode is the professional layer around them: the bounty lawyers, the litigation funders, the people who make filing tractable for someone who has never done it. That layer is funded by contingency and it does starve. So the effect is real but it is second-order: not that nobody notices, but that noticing becomes harder to act on. I am fairly comfortable here, and I would not be if the design had put professionals rather than colleagues in the detector’s chair.
What this changes
Three things, in descending order of how much they should bother anyone.
The insurance leg needs a non-experience price anchor, and the site does not currently specify one. This belongs on page 08 and in the statute sketch, not in a note, and I have not written it yet.
Exit is an interior solution, not a corner one. Page 09 should say where it expects the exodus to stop and what the frontier labour market looks like at that wage. The answer is probably favourable, because sits in the exponent, but the page currently does not ask.
The equilibrium is dynamic, and the site keeps describing it as a resting place. Between the softening premium here and the repeal risk, success and redundancy being indistinguishable from outside, the quiet world contains two independent mechanisms that erode it, both powered by the quiet itself. A regime like this is not something you install. It is something that has to be maintained against its own success, and anything that gets drafted should have that assumption in it from the first version rather than discovering it in year eight.
None of that touches the central claim. Repricing a labour market is still an easier problem than specifying values into an optimiser, and every difficulty above is a difficulty of keeping a price honest, which is a problem with a literature, a profession and two centuries of practice behind it. But “easier than alignment” is a low bar to clear, and I have been treating it as though it were the finish line.