Extinction Bounties

Policy-based deterrence for the 21st century.

The Sequence · Part II · What To Do

08. The Insurer

Deterrence straight from the underwriter's office.

Last revised 11 September 2026 · alpha

The bounty is the part everyone reacts to. It is vivid, it is a little lurid, and it is what makes people describe this proposal as a scheme for turning colleagues against each other.

It is also the half that almost never runs. The half that runs every day is the premium.

Why the fine has to be insured at all

Start with the version of the mechanism that has no insurer in it. A statute defines a class of dangerous work, attaches a very large penalty, and pays a share of it to whoever brings the evidence. A researcher who crosses the line owes, say, ten years of total compensation.

That fine is not real. Almost nobody has that money.

This has a name in the law and economics literature, and I should use it rather than reinvent it: the judgment-proof problem . Someone who cannot pay what they have been found liable for does not merely underpay after the fact — they take too little care beforehand, because the part of the harm that exceeds their assets never enters their decision at all. They are not facing the penalty the statute wrote down. They are facing their own net worth, which is a different and much smaller number.

The structural consequence is that the penalty schedule goes flat above the ruin threshold. Everything you have is the same amount whether the violation was marginal or flagrant, so nothing is deterred at the margin. The standard conclusion from there is that you use the fine to its maximum feasible extent and then reach for imprisonment, the sanction nobody is judgment proof against. That reach is the thing this page is trying to avoid needing.

Where the judgment-proof result comes from

Steven Shavell, The Judgment Proof Problem (1986), establishes the ex ante care result: liability beyond a defendant’s assets does not influence behaviour, so judgment-proof parties take systematically too little care.

Polinsky and Shavell work the optimal enforcement problem and conclude that the fine should be used to its maximum feasible extent before imprisonment is added — imprisonment being costly to society in a way fines are not, but reachable when a fine is not.

And the behavior a flat schedule produces is exactly the behavior you least want. Someone facing certain ruin has no incentive to cooperate, to document honestly, to settle, or to stay in the jurisdiction. They have every incentive to destroy the evidence and fight to the end, because there is no outcome short of the end that differs from it. I have worked through the ugly cases in a separate note .

This population is judgment proof in an unusually sharp way

The judgment-proof literature is mostly written about defendants who are poor and will stay poor: the thinly capitalized firm, the driver with no assets and no policy. The people this proposal is aimed at are a different case, and I think a worse one.

A frontier researcher is not poor. They are pre-rich. A great many of them are early in a career that will pay extremely well, and the defining financial fact about an early-career high earner is that lifetime earnings are enormous while net worth right now is small — savings barely started, equity unvested or underwater, and whatever cash exists already committed to the rent in one of the most expensive metros on earth. The gap between what they will earn and what they own is the widest it will ever be, and it is widest at exactly the age at which they are doing the work.

Now peg the fine at ten years of total compensation , the roughly $2.25 million from page 07. For this specific population that number is almost perfectly calibrated to be uncollectable. They could pay it over a career. They cannot pay a tenth of it this afternoon. And Shavell’s point is that the only number that does any deterring is the one they face at the moment of the decision — so the statute writes $2.25 million, the researcher faces whatever is in the brokerage account, and the difference between those two figures is deterrence that was legislated and never delivered.

So the population most able to cause the harm is the population least reachable by a fine. That is not a mild version of the judgment-proof problem. It is close to the worst case for it.

What makes this an argument for the insurer rather than a fatal objection is that the same facts which make the fine uncollectable make the premium easy. Insurance is the instrument that converts future income into present liability coverage. The fine has to be paid out of savings, which they do not have. The premium is paid out of salary, which they emphatically do. Everything that makes them judgment proof — young, liquid-poor, income-rich, with decades of earnings ahead — is exactly what makes them a good insurance risk and a bad defendant.

The alternatives for reaching those future earnings are all worse. Wage garnishment is slow, bounded by jurisdiction, and defeated by the move abroad this page has already worried about. Prison is the Polinsky and Shavell fallback and the thing I am trying not to need. A premium is garnishment collected in advance, from someone who has not been caught, for a harm that has not happened, and it is the only one of the three that does any work before the fact.

One catch I will not pretend away: if the lab reimburses the premium, the individual gradient is blunted, and labs would obviously try. The partial answer is that the rate is still individually priced, so a lab that reimburses pays more for the researchers who are riskier and for the projects that are more dangerous — the signal survives, it just relocates into the wage bill, which is the price system doing the thing the previous section described. Whether that is good enough is a real question and it is page 12’s , not mine to wave off here.

Insurance is what makes the number real. It converts a liability that cannot be paid into a premium that can be, and in doing so restores the gradient Shavell showed was missing: the more dangerous your work looks, the more you pay, continuously, in amounts calibrated to how dangerous it looks rather than to how much you happen to own.

This is not a clever discovery of mine — it is the standard response to the standard problem, which is why the Weil paper further down this page is called Overcoming Judgment-Proofness and why Cristian Trout’s nuclear-precedent paper recommends mandatory insurance for frontier developers — to “overcome developers’ judgment-proofness, mitigate winner’s curse dynamics, and leverage insurers’ quasi-regulatory abilities.” Price-Anderson has been doing the first of those for American nuclear power since 1957.

The underwriter has money on it

Here is what an insurer does that no regulator can.

A regulator inspects you periodically, from a fixed budget, on a schedule you can anticipate, using criteria written some years ago by people who do not lose anything personally if they are wrong. Their attention is a scarce public resource allocated politically. They are also, structurally, capturable — the industry they supervise is the only place to hire expertise from and the most attractive place for their own staff to go next.

An underwriter is exposed on every policy they write. They are not assessing whether you comply with a rule; they are estimating what you will cost them, and they are doing it with their own balance sheet as the stake. That estimate gets revised whenever they learn something new about you, and the revision arrives as a number you have to pay. Underwriting margins are thin — single digits is the usual figure — which means even a small revision in expected loss is not something an insurer can absorb quietly out of politeness.

The consequence is that the insurer polices you continuously and specifically, in a way no inspection regime does. They will want to know your headcount, what your team is actually working on, how close it runs to the statutory line, what your logging looks like, whether you can demonstrate what you trained and on what. Not because a rule requires them to ask, but because the answers move the price they are willing to quote.

The premium is a price on your colleagues’ silence

There is a specific thing the premium is measuring, and it connects this page directly to the arithmetic in the previous one .

An insurer pricing this risk is not trying to estimate the probability of a catastrophe. They are estimating the probability of a claim — that someone produces evidence and collects. Strip a premium down and it is the size of the bounty multiplied by the chance somebody claims it, plus the insurer’s own margin. The margin is the uninteresting term. The chance somebody claims is exactly the quantity page 07 was computing : it rises with the number of people who know.

The premium, decomposed

For a bounty BB, the premium π\pi is roughly

π    qBexpected payout  +  λthe insurer’s cut\pi \;\approx\; \underbrace{q \cdot B}_{\text{expected payout}} \;+\; \underbrace{\lambda}_{\text{the insurer's cut}}

where λ\lambda is what the trade calls the loading — the margin added on top of expected loss to cover expenses, the cost of holding capital against the risk, and profit — and qq is page 07’s 1pn1 - p^{\,n} , the chance a claim lands in the policy period.

Everything interesting is in qq. λ\lambda is why insurance costs more than it pays out, and it is the same term in every line of business.

So the premium is, quite literally, a continuously updated market price on the silence of the people you work with. Hire twenty more people onto the project and your premium moves, because the number of people who know moved. Have a bad quarter for morale and it moves again. You are being quoted a daily figure for how much your colleagues can be trusted, by a party with money riding on the estimate.

I find that a stranger and more interesting object than the bounty itself.

Deterrence arrives as a bill

Which brings me to the part I actually care about.

Criminal enforcement deters by firing. It needs arrests, prosecutions, and publicity; the deterrent effect is downstream of somebody visibly being destroyed, and when the machinery goes quiet the deterrence decays. That is an unpleasant way to run a system, and it is unpleasant in proportion to how serious the offense is.

Insurance deters by pricing. The deterrent shows up as a line item in a budget before anything has happened to anyone. A lab whose premiums have tripled has been told something important, and told it without any person being raided, charged, tried, or ruined. The mechanism can be working at full strength, doing everything it was designed to do, in a world where no bounty has ever been paid and no one has ever been prosecuted.

That is the success condition, and it is worth being explicit that it is a strange one — I have argued it on its own terms separately : a policy whose ideal outcome is that its most dramatic component never operates. Page 09 takes this further — no prosecutions is a win, not a failure — but the reason it can be a win is entirely on this page. Without the premium there is nothing to deter with between prosecutions.

Why a price and not a rule

I want to be explicit about the machinery this is leaning on, because I do not think it is intuitive and I think it is the actual load-bearing claim of the page.

The thing you would most want to know about a frontier lab — how close is this project, right now, to doing something catastrophic — is not the kind of fact that can be collected. It is not in a filing. It exists as what Hayek called the knowledge of the particular circumstances of time and place : spread across a few hundred people who each hold a fragment. Someone knows the eval was rerun until it passed. Someone knows the safety case was written after the decision it justifies. Someone knows the colleague who kept objecting has stopped objecting. None of them holds the whole picture, and every one of them holds a piece no inspector could extract on a schedule.

Centralizing that knowledge is precisely the thing Hayek argued cannot be done — not because regulators are lazy or captured, but because the knowledge is perishable, tacit, and partly about the holder’s own judgment of people he works with. What a reporting requirement gets you instead is the compliance artifact: a description of the situation as of a quarter ago, written by people who knew it would be read.

Hayek’s move was to stop trying to transport the knowledge and to transport a number instead. A price, on his account, is a piece of telecommunications equipment: it lets people who know nothing about the underlying situation act correctly on it anyway, by watching a pointer move.

Hayek on prices as telecommunications

It is more than a metaphor to describe the price system as a kind of machinery for registering change, or a system of telecommunications which enables individual producers to watch merely the movement of a few pointers, as an engineer might watch the hands of a few dials, in order to adjust their activities to changes of which they may never know more than is reflected in the price movement.

And on what that buys:

The marvel is that in a case like that of a scarcity of one raw material, without an order being issued, without more than perhaps a handful of people knowing the cause, tens of thousands of people whose identity could not be ascertained by months of investigation, are made to use the material or its products more sparingly.

A premium is such a pointer. Nobody has to publish what they know. Nobody has to be believed. No regulator has to become competent at evaluating training runs on a six-week cycle. The number moves, and a board that understands nothing whatsoever about mechanistic interpretability understands perfectly well that the number went up. Substitute organizational recklessness for Hayek’s scarce raw material and that is the whole proposal. This is also why page 09 can treat zero prosecutions as the win condition. A price does not need anyone to be punished in order to be informative, which is not true of any enforcement mechanism that works by making examples of people.

Where the analogy gives out

Three places, and I would rather state them than have them found.

An underwriter is not a market. Hayek’s prices emerge from many dispersed parties each trading on knowledge only they have. A premium is set by a handful of underwriters running a model. That is closer to central planning with better incentives than it is to a price system, and I should not dress it up as the latter. What rescues it partially is that the underwriter is financially exposed to being wrong, which no regulator is, and that competing insurers disagreeing about the same lab is itself a signal.

The insider’s knowledge needs a channel. In Hayek the man on the spot acts on what he knows by trading on it. Here the researcher’s private knowledge reaches the premium only through an underwriter’s guess about pp. The thing that actually lets him act on what he knows is the bounty. That is the real reason the bounty and the insurance are not two policies stapled together: the bounty is the trading mechanism that makes the premium an aggregator of knowledge rather than an actuary’s opinion. The next section is about the second channel this opens, which I think is the most underrated part of the whole design.

Hayek’s other warning cuts against me. The same author wrote The Pretence of Knowledge , which is largely about the damage done by people who dress a guess in mathematics because the mathematics is the part that looks like knowledge. Anyone writing q=1pnq = 1 - p^{\,n} and then reasoning about the answer is doing something in that family. I think the defense is that the arithmetic on these pages is there to show the shape of the thing — that qq climbs steeply in nn, which is the only property the argument uses — and that nobody in the actual mechanism has to estimate pp at all. The insurer does, and eats the error if it is wrong. But I would rather concede that the numbers are illustrative than have someone discover it and conclude the rest was sold the same way.

Telling your own insurer

Here is the part I did not see when I first wrote this down, and I now think it may be the most useful thing in the mechanism.

Consider a researcher who is not blowing any whistles. They have a bad feeling. The evals are being rerun more than they used to be. The safety case arrived suspiciously soon after the decision it justifies. Nothing they could put in a filing, nothing that would survive a lawyer, just the accumulated unease of someone who can see the work up close.

Mostly that person has two options: say nothing, or destroy their career. Whistleblowing is a one-shot, public, irreversible act that ends your employment and usually your industry, and it requires you to be certain in advance about something you are merely uneasy about. The overwhelming majority of people with a bad feeling take option one, and the reason is not cowardice. It is that the action space has a hole in the middle of it.

I should not overstate that, because people closer to the problem than me have noticed the same hole and started filling it. Karl Koch’s AI Whistleblower Initiative runs a programme called Third Opinion , and the name is the thesis: it lets an insider find out anonymously whether what they are worried about is real before they have to do anything irreversible about it.

How Third Opinion actually works

An insider contacts the AI Whistleblower Initiative — formerly OAISIS — anonymously through an open-source Tor tool. Their concern gets workshopped into a well-posed question, and independent experts answer it without the insider ever disclosing who they are or who they work for. Only if the concern turns out to be justified does it escalate to pro bono counsel, where privilege attaches.

Koch’s diagnosis of why the existing options fail is worth sitting with: insiders do not approach whistleblower lawyers because even experienced ones may lack the technical knowledge to evaluate the concern, so the insider cannot tell whether they have seen something real before committing to an act that cannot be walked back.

That is a genuine third option and I do not want to write it out of the picture. But notice what it is. Third Opinion is diagnostic. It exists to help you determine whether your unease is well-founded and, if it is, to jump more safely. It does not give you something to do instead of jumping, it is not a repeated transaction, nothing about it is priced, and — the part that matters most — AIWI has no leverage over your employer. A lawyer can protect you. Only a counterparty who can reprice the firm or decline to cover it can change what the firm does.

So the hole is narrower than I first wrote, and the remaining shape of it is specific: there is still nothing a researcher can do that is continuous, repeatable, financially meaningful, and aimed at the lab rather than at their own legal exposure.

Now give them an insurer. They can pick up the phone to their own underwriter and say: I think the thing I am working on has gotten more dangerous, here is roughly why. That is a private, repeatable, survivable act. It does not end anything. It can be partial — you can say the small version this quarter and the larger version next quarter, and the response you get is a number rather than a verdict. Disclosure becomes something you can titrate, which is precisely what you cannot do with a newspaper or a regulator.

And there are real reasons to do it, not just permission to:

You may already be obliged. Insurance contracts run on utmost good faith — the duty to disclose material facts, with nondisclosure grounds for voiding the policy. A researcher who knew the work had changed character and said nothing risks discovering they were uninsured at the exact moment it mattered. That is a stick, and it operates without anyone having to be brave.

Silence gets priced anyway. This is the unraveling result , from Milgrom and from Grossman, both in 1981: where disclosure is possible and cheap, withholding is read as bad news, so the best of the silent have an incentive to separate themselves by speaking, and the logic iterates until silence means the worst. Applied here, an underwriter facing a researcher who volunteers nothing will assume the pooled worst case and charge for it. The researcher whose project really is fine has a straightforward financial reason to prove it. Nobody is compelled and disclosure happens anyway, which is the same trick the premium pulls on the lab.

The insurer has leverage you do not. A junior researcher cannot change what a frontier lab does. An underwriter who can reprice the whole firm, demand controls as a condition of coverage, or decline to renew, very much can. Telling your insurer is not just unburdening yourself; it is handing your observation to the only party in the picture with both the standing to act on it and money riding on getting it right.

It is a defense. Early disclosure and documented remediation is protection against somebody else’s claim landing on you later. The researcher who raised it in writing eighteen months ago is in a different position from the one who did not.

Then the market does what markets do. Brokers appear who specialize in this line. Underwriters hire people who can actually read a training run — which is the same shortage Koch identifies, approached from the other side. AIWI solves the missing-technical-competence problem with volunteers and pro bono time. An insurer would have to solve it with a profit and loss statement, and I expect that to scale further, though I note it is also the less admirable of the two motives. Coverage speciates — project-specific riders, different terms for people who accept telemetry, cheaper rates for researchers who can demonstrate they were the ones objecting. I am speculating about the details, but the general direction is the ordinary history of how insurance markets mature, and it produces a profession whose job is to be technically competent about frontier AI risk and financially punished for getting it wrong. We do not currently have one of those.

The part I want to underline, because it is the answer to the objection in the previous section: none of this works without the bounty. Information about a lab’s recklessness is worth nothing to an insurer if no claim can ever be brought on it — there is nothing to price. The bounty is what makes the information financially meaningful, and the insurance is what makes it disclosable without career suicide. Either alone is inert. Together they are the channel Hayek’s story needs and the reason I keep insisting these are one mechanism.

Three ways this goes wrong

There is no insurer-insured privilege. This is the serious one. Neither federal common law nor most states recognize a general privilege covering communications between an insurer and its policyholder; as one practitioner summary puts it, there is “no insurer-insured privilege per se” , and such protection as exists depends on state law and on the communication being routed toward defense counsel. So the researcher who calls their underwriter has created a documented record of their own awareness, held by a third party who can be subpoenaed and whose interests diverge from theirs the moment a claim is filed. As the law stands, the honest advice to that researcher is to say nothing directly.

There is a workaround, and AIWI is already using it: route the disclosure through counsel, where privilege does attach. A researcher who tells their lawyer, who tells the underwriter what needs telling, gets most of the channel with the protection intact. That is clumsy, it is expensive, and it puts a solicitor in the middle of what ought to be a phone call — but it means the thing is possible today rather than blocked pending legislation, and it is roughly how large commercial insureds already handle sensitive disclosures.

The clean version still needs a privilege carved for the purpose, and I do not know whether that is politically achievable. It is the largest piece of legal construction this proposal needs, and I had not registered that until I worked through this section.

The insurer becomes an informant network. Employees confidentially reporting on their employers to a private company with no constitutional constraints, a commercial interest in collecting more than it needs, and no obligation to anyone but its shareholders. This is the same worry as the monitoring section below, one degree worse, and I do not have an answer to it.

The insurer can turn on its own customer. If the policy excludes deliberate violations — and insurance law makes it very hard not to exclude them — then an insurer that discovers deliberate conduct owes nothing and can claim the bounty itself. I have written that case up separately , because I think it is less dangerous than it first appears but more dangerous than I first thought.

Who else has proposed something like this

The mechanism underneath all of this is Robin Hanson’s, from Privately Enforced & Punished Crime in 2018. I did not invent fine-insured bounties and have never claimed to. What I did was point them at AI, and I want to be accurate about when, because the legal literature has since converged on something adjacent and it would be easy to read this page as downstream of it.

It is not — the gap is at least a year and a half in the other direction.

The dates, since priority runs downhill toward credentials

The original Bounties on AI Researchers went up on my old site, virtual-instinct.xyz; the Wayback capture is dated 9 July 2022, and that capture is already of an older post from my Twitter days. I republished it with commentary in March 2023.

Gabriel Weil’s Closing the AI Accountability Gap gives a writing date of 13 January 2024 and was posted to SSRN that month.

I am not claiming Weil read me — I am confident he did not, and convergence is the expected outcome when two people stare at the same judgment-proofness problem. I raise the dates only because he is the credentialed one, and priority claims run downhill toward credentials unless somebody writes the dates down.

The substantive point is that we did not arrive at the same mechanism anyway.

Gabriel Weil’s framework treats frontier training as an abnormally dangerous activity triggering strict liability, with mandatory insurance scaled to the worst harms a system could plausibly cause; his Overcoming Judgment-Proofness works the same problem this page opened with, as law and economics rather than as intuition. Page 11 sets his proposal out properly and compares it to this one, because comparison is that page’s job.

The one difference that belongs here, because it is a fact about insurance rather than about rival proposals, is the party being insured. They insure the firm against third-party harm. This insures the individual against a statutory bounty. That is not a detail, and the next section is why.

Why the individual and not the firm

If there is one design choice on this site I would defend hardest, it is this one, and I do not think I have made the case for it forcefully enough.

Firm-level and individual-level insurance sound like two implementations of the same idea. With respect to the thing this mechanism is actually trying to do, they are opposites.

Firm-level insurance pools the conspiracy. Make the company the insured party and every employee’s financial interest points in the same direction the company’s does: no claim, nothing found, nothing paid. The policy is a shared stake in nothing being discovered. You have not disturbed the wall of silence — you have bought it a group discount. Whatever else firm-level cover achieves, and it achieves real things, it gives no individual in that building a single reason to behave differently tomorrow morning.

Individual-level insurance divides it. My premium is priced off my own exposure, and my exposure depends on nn — on how many people are in the room and what they are doing. Your recklessness raises my bill. Not after a catastrophe, not if a court ever finds anything: at the next renewal. That is a standing financial interest I hold that runs directly against your ability to keep something quiet, and it exists before anything has gone wrong. The insurance is not merely a way of paying the fine. It is the thing that puts the hundred people in the building on non-identical sides.

Three consequences follow, and each of them is load-bearing.

The arithmetic requires individuals. q=1pnq = 1 - p^{\,n} describes nn people each independently deciding whether to stay quiet. A firm-wide policy has one decision-maker and therefore one pp. You cannot extract superlinear scaling in headcount from a single contract signed by a general counsel — there is nothing in it that varies with nn, because the whole firm is one insured.

Only an individual policy creates an exit. Page 09 says the goal is to make dangerous work a worse career than the alternative. A policy that follows the person does that automatically: change employers and your rate reprices, so the cost of staying somewhere reckless is a number you personally see, personally pay, and can personally escape by leaving. Firm cover is invisible to the employee and creates no pressure to go anywhere.

Firm solvency does not cure individual judgment-proofness. A well-capitalised lab standing behind a corporate policy does nothing about the researcher who personally owes $2.25 million and personally cannot pay it. The problem this page opened with is an individual problem and it wants an individual instrument.

Doctors already do this

The reflexive objection is that it is unreasonable to ask individuals to carry career-scale personal liability, and that no profession would accept it.

An entire profession accepts it, and has for decades. Malpractice liability attaches to the individual physician, not only to the hospital. Cover is underwritten individually — on specialty, geography, procedure mix, and personal claims history — and the resulting variation is enormous: something like a twentyfold spread in the annual personal cost of practising, driven by how dangerous your particular work is.

How large the spread actually is

Jena and colleagues, looking at nearly 41,000 physicians over fifteen years, found that 19.1% of neurosurgeons face a claim in a given year against 2.6% of psychiatrists , and that by age 65 a claim has been made against 99% of physicians in high-risk specialties and 75% in low-risk ones.

Premiums track it: roughly $7,500 to $12,000 a year in the low-risk specialties against $150,000 or more for neurosurgery in litigious jurisdictions, with an OB-GYN inside Cook County paying something like double one outside it.

Roughly four in five malpractice claims close with no payment at all.

Sit with that for a moment, because it is the whole argument in miniature. A twentyfold spread in the personal cost of practising, varying by what you do and where you do it, recalculated annually, and nobody regards it as scandalous. It is simply understood that some work is more dangerous than other work and that the person doing it carries a correspondingly larger number.

Two further details transfer directly. The first is that compensation adjusts — high-premium specialties command higher pay, which is precisely the wage pass-through page 09 predicts, observed in a real labour market for fifty years. The institution ends up paying, through the wage bill, without being the insured party. That is the thing I am claiming would happen, already happening somewhere else.

The second is that most claims fail. The insurer defends, four in five close with no payment, and the profession does not collapse into extortion — which is the empirical answer to the worry that a bounty regime would drown in opportunistic filings.

Where the comparison breaks

I do not want to oversell it.

Employed physicians are increasingly covered by their institutions rather than buying their own policies, so the pure “every professional holds their own” picture is less true than it was. What survives is the part I need: liability attaches personally, rating is individual, and tail coverage is the departing doctor’s problem. But if the objection is that institutions absorb these markets over time, the objection has evidence behind it and I should say so.

Malpractice is ex post and negligence-based. A patient is harmed, sues, and a court assesses conduct against a standard of care. A bounty fires ex ante with no victim at all. What transfers is the insurance architecture, not the theory of liability, and anyone reasoning from one to the other should be careful about which they are borrowing.

Doctors generate loss data. A hundred thousand claims a year make an actuary’s job possible. Bounty claims would be rare, which is the pricing problem the insurability note ends on.

Page 11 does the comparison with Weil properly. I have laboured it here because the choice of insured party is the hinge the entire mechanism turns on, and it is easy to read past as a technical detail.

What the insurer will demand, and where that goes wrong

An insurer with money at stake will want monitoring — logging, telemetry, audit rights, some ability to verify that you are doing what you said. They will offer better terms to anyone who accepts it, which means the market generates a transparency regime nobody had to legislate. There is also a useful adverse-selection effect: refusing to be monitored is itself information, and it gets priced.

That is a feature right up until it becomes a surveillance apparatus operated by private companies over the working lives of researchers, with no constitutional constraints on it and a commercial incentive to collect more than it needs. I do not think that concern is overblown and I am not going to answer it here; page 12 is where the objections live and this one belongs near the top of them.

On the two questions everyone asks next — who would write this coverage at all, and whether it should be voluntary or compulsory — I have separate notes, on insuring a criminal and on the voluntary-or-mandatory fork . The second is genuinely unresolved in my own head and I would rather leave it that way here than pretend otherwise.

The counter-case

The strongest objection to this page is not a thought experiment. It is an existing market that was supposed to work like this and largely did not.

Daniel Schwarcz and Josephine Wolff’s The Limits of Regulating AI Safety Through Liability and Insurance: Lessons From Cybersecurity makes the case. Cyber insurance is a mature, competitive market with exactly the incentives described above, and it has been a disappointment as a safety mechanism. Their diagnosis: without reliable historical loss data, without a way to assess a particular firm’s risk level, without knowing which safeguards actually work, and with catastrophic tails in play, insurers end up helping customers limit liability rather than reducing anyone’s exposure. Every one of those conditions holds for AI, and several hold worse — the technology moves faster, the systems are more opaque, and they are threaded through everything. Their recommendation is that mandated, standardized safety-data collection has to come first.

That is a serious objection and it is aimed squarely at the mechanism on this page. It is also, I think, much harder on the firm-level proposals than on this one, for reasons page 11 works out .

Here is why I think it lands more softly on this proposal, and I want to be careful because this is the load-bearing reply. It gets made here, once, and pages 11 and 12 refer back to it rather than restating it.

Every obstacle they identify is an obstacle to pricing harm. The premium here does not price harm.

Look at what their four difficulties are difficulties about: estimating how likely an AI system is to cause a loss, how large, and to whom, in a technology that changes faster than the model and cannot be disentangled from the business around it. Those really are close to insuperable, and I do not think anyone can price the probability of a catastrophe.

An underwriter in this design is never asked to. The bounty is not a catastrophe; it is a defined statutory sum, triggered by an evidentiary event, adjudicated in a court. What has to be modelled is claim frequency — how likely it is that somebody in this building produces evidence and files — which is what directors-and-officers and professional-indemnity underwriters do every day, on risks that are also opaque, also adversarial, and also short of clean loss data. The inputs are unusually legible besides. Headcount is legible. Tenure is legible. How close the work runs to a defined statutory line is legible, because a statute defined it. Compare that to asking whether a model will cause a systemic failure in four years.

That reframing also dissolves the first-decade objection, which was the part I was most worried about. There is no loss history for AI catastrophes and there never will be until it is too late to matter. But the quantity actually being priced is the rate at which insiders disclose misconduct, and that has decades of data behind it from completely unrelated domains — SEC whistleblower filings per firm-year, qui tam filings, employment-dispute and internal-escalation rates. Organisational leakage is not a novel phenomenon. An actuary pricing it has considerably more to work with on day one than the objection assumes.

Why cyber is the wrong analogy, in two specific ways

The cybersecurity comparison still deserves respect, because it is a real failure of a real market rather than a thought experiment. But the two properties that broke it are the two this design inverts.

A cyber claim has no insider claimant. It is filed by the victim firm against its own policy, after an attack, by the very people whose security posture is at issue. Nobody inside that firm is paid for revealing that its security was negligent, so the insurer’s information problem is exactly as bad as its own investigative capacity. Here the claimant is an insider, and the information the underwriter cannot gather is supplied by people paid to supply it.

And a cyber trigger is partly exogenous. Whether you are breached depends on who decided to attack you, which drowns the signal running from conduct to claim in noise about attacker behaviour. Here the trigger is the insured’s own conduct against a line a statute drew.

Cyber insurance underperformed as a regulator in substantial part because nobody had any reason to tell the insurer anything true. That is the one feature this design does not share.

What does not survive the objection

Two things, and they are not trivial.

Their attack on insurance as a non-price regulator lands squarely. The monitoring story below — audit rights, conditions on coverage, controls demanded as a term of the policy — is precisely what cyber insurers attempted, and it underdelivered. I hold that part of this page much more loosely than the pricing part, and the honest position is that this mechanism should be expected to work through the premium rather than through the underwriter’s clipboard.

And a brand new statute has no claim history at all, so the first years of pricing will be guesswork and wrong in one direction or the other for a while. Their observation that insurers drift toward helping clients limit liability rather than avoid it applies to any policy covering defence costs, which this one would. Neither is fatal. Both mean the mechanism would be badly calibrated early, and being badly calibrated early is how most regulatory schemes fail permanently.

Their remedy is also complementary rather than rival. Mandated, standardised safety-data collection is the best underwriting input anyone could ask for. Enacting their proposal first would make mine work better, which is a strange thing to be able to say about your strongest critic.

To cite this page: Andrew Quinn, "08. The Insurer." Extinction Bounties, last revised 2026-09-11. https://extinction-bounties.com/sequence/08-the-insurer/

Policy-research disclaimer

Extinction Bounties publishes theoretical economic and legal mechanisms intended to stimulate scholarly and public debate on catastrophic-risk governance. The site offers policy analysis and advocacy only in the sense of outlining possible legislative or contractual frameworks.

No legal or financial advice

Nothing here should be treated as a substitute for qualified legal counsel, financial due diligence, or regulatory guidance. Readers remain responsible for ensuring their actions comply with the laws and professional standards of their own jurisdictions.

Exploratory and personal views

All scenarios, numerical examples and opinions are research hypotheses presented by the author in a personal capacity. They do not represent the views of the author's employer, funding bodies, or any governmental authority.

Implementation caveats

Any real-world adoption of these ideas would require democratic deliberation, statutory authority, and robust safeguards against misuse. References to enforcement, penalties, or "bounties" are illustrative models, not instructions or invitations to engage in private policing or unlawful conduct. Nothing here is directed at any identifiable individual — see non-targeting.

No warranty and limited liability

Content is provided "as is" without warranty of completeness or accuracy; the author disclaims liability for losses arising from reliance on this material.

By continuing beyond this notice you acknowledge that you have read, understood, and accepted these conditions.