Extinction Bounties — Policy-based deterrence for the 21st century

It is easier to shift the Nash equilibrium of working on AI in the first place than it is to create safe AI.

This site makes one argument in two halves. Part I is why I think a world that builds sufficiently capable AI is likely to be a world with nobody home in it — a case that runs through consciousness, other minds, and what utility functions actually select for. Part II is what I think can be done about it, which is where most of the work is.

The short version of Part II: attach a large statutory penalty to a narrow class of frontier activity, pay a bounty out of that penalty to anyone on earth who brings evidence leading to a conviction, and require the individual researcher — not the firm — to carry liability insurance against it. The insurance is the load-bearing part. A fine nobody can pay deters nobody, and a premium is a price on your colleagues’ silence: firm cover pools the conspiracy, individual cover divides it. Enforcement funds itself out of the penalties, so no legislature has to find the money, and adoption needs a number and a court rather than a treaty body. If it works, almost nothing happens — the deterrence is the product, and that makes it hard to measure.

That is the whole mechanism. The two-minute version fills it in; the sequence below argues it.

If none of the AI-risk framing is familiar, aisafety.info is the primer I actually recommend, and I would rather you read it than my compressed version of it.

A note on provenance: this is a signed rewrite of the anonymous extinction-bounties.github.io , and the original 2022 post is at andrew-quinn.me/ai-bounties . This site is canonical. Nothing here is directed at anyone in particular — see non-targeting .

The Sequence — one argument, thirteen pages

Part I · Why Bother

01. The Only Thing You Know Is Conscious
The one fact about qualia you have direct access to is your own, and you cannot get behind that fact to check anyone else's.
alpha
02. A Best Lower Bound on Other Minds
A similarity-weighted, pluralist prior over who is a moral patient — the best answer I have found to a question that admits no proof.
alpha
03. Competence Does Not Require Qualia
The successionist error: treating capability as evidence of moral patienthood. It is not, and people are strangely reluctant to say so.
alpha
04. The Universe With Nobody Home
Not suffering, not human extinction as such: a cosmos in which nothing experiences anything, and which therefore has no moral relevance at all.
alpha
05. Almost All Utility Functions Are Irrational
Instrumental convergence, Bostrom and Yudkowsky, and an intuition about measure: the aligned targets are a vanishingly thin set.
alpha

Part II · What To Do

06. The Hook
A thousand-dollar littering fine, a fifty-dollar camera, and the small market that grows up around them.
alpha
07. The Gist
Hanson's fine-insured bounties in general form, and the conspiracy arithmetic that makes silence expensive.
alpha
08. The Insurer
The insurance market is the part that does the actual deterring, and it does it before anyone is ever charged.
alpha
09. The Goal
The target is not punishment and not even prevention of any particular project. It is to make the marginal researcher take a different job.
alpha
10. Making It Global
A mechanism that stops at a border relocates the risk instead of reducing it. What adoption would actually have to look like, and what it would still fail to reach.
alpha
11. Compare and Contrast
How this proposal stacks up against the other things people suggest, including on the axes where it loses badly.
alpha
12. The Catches
The strongest objections to the mechanism, collected in one place, stated at full strength, and answered only where I can answer them.
alpha
13. Next Steps
What a serious first version looks like, starting somewhere that is not AI at all.
alpha

Recent writing

When Your Insurer Collects the Bounty
An insurer that turns in its own policyholder pays the fine and collects the bounty, netting roughly nothing. That arithmetic breaks the moment the policy excludes willful violations — which insurance law all but requires.
alpha 11 September 2026
What Bounties Would Have Done About Hugging Face
The first confirmed end-to-end intrusion by an autonomous agent against AI infrastructure. Walking the incident backwards to find the decision a liability regime would actually have priced.
alpha 11 September 2026
The Frontier Nobody Is Pricing
Gain-of-function work has the exact shape fine-insured bounties were designed for: few actors, catastrophic tails, low detectability, and colleagues who know.
alpha 11 September 2026

Policy-research disclaimer

Extinction Bounties publishes theoretical economic and legal mechanisms intended to stimulate scholarly and public debate on catastrophic-risk governance. The site offers policy analysis and advocacy only in the sense of outlining possible legislative or contractual frameworks.

No legal or financial advice

Nothing here should be treated as a substitute for qualified legal counsel, financial due diligence, or regulatory guidance. Readers remain responsible for ensuring their actions comply with the laws and professional standards of their own jurisdictions.

Exploratory and personal views

All scenarios, numerical examples and opinions are research hypotheses presented by the author in a personal capacity. They do not represent the views of the author's employer, funding bodies, or any governmental authority.

Implementation caveats

Any real-world adoption of these ideas would require democratic deliberation, statutory authority, and robust safeguards against misuse. References to enforcement, penalties, or "bounties" are illustrative models, not instructions or invitations to engage in private policing or unlawful conduct. Nothing here is directed at any identifiable individual — see non-targeting.

No warranty and limited liability

Content is provided "as is" without warranty of completeness or accuracy; the author disclaims liability for losses arising from reliance on this material.

By continuing beyond this notice you acknowledge that you have read, understood, and accepted these conditions.