Extinction Bounties

Policy-based deterrence for the 21st century.

Developing frontier AI risks human extinction. It should be made illegal. Extinction bounties are one good way to enforce that law.

Play the game: watch a bounty system work →

Read the five-minute pitch →

How it works, in four steps

  1. Dangerous AI development is made illegal. Pushing frontier AI capabilities further carries a hefty fine.
  2. Turn them in and you get paid a bounty. Your bounty is the fine, less court processing fees. Enforcement pays for itself. Anyone with enough evidence can claim a bounty, ideally anywhere in the world.
  3. Insurance covers the fine, at a premium. Software developers start to carry their own insurance against the risk of such fines. Like car insurance, careful devs pay little and reckless ones pay a lot. The insurer has every reason to find out which is which.
  4. The dangerous work never starts. Success looks like nothing happening. Frontier AI development rapidly chills worldwide. A few projects get quietly shelved. A few people taking the other job offer. A world saved.
Policy-research disclaimer

Extinction Bounties publishes theoretical economic and legal mechanisms intended to stimulate scholarly and public debate on catastrophic-risk governance. The site offers policy analysis and advocacy only in the sense of outlining possible legislative or contractual frameworks.

No legal or financial advice

Nothing here should be treated as a substitute for qualified legal counsel, financial due diligence, or regulatory guidance. Readers remain responsible for ensuring their actions comply with the laws and professional standards of their own jurisdictions.

Exploratory and personal views

All scenarios, numerical examples and opinions are research hypotheses presented by the author in a personal capacity. They do not represent the views of the author's employer, funding bodies, or any governmental authority.

Implementation caveats

Any real-world adoption of these ideas would require democratic deliberation, statutory authority, and robust safeguards against misuse. References to enforcement, penalties, or "bounties" are illustrative models, not instructions or invitations to engage in private policing or unlawful conduct. Nothing here is an accusation against, or a call to act against, any person or organisation; see non-targeting.

No warranty and limited liability

Content is provided "as is" without warranty of completeness or accuracy; the author disclaims liability for losses arising from reliance on this material.

By continuing beyond this notice you acknowledge that you have read, understood, and accepted these conditions.


The Sequence: one argument, thirteen pages

Part I · Why Bother

  1. 01. The Only Thing You Know Is Conscious

    The one fact about qualia you have direct access to is your own, and you cannot get behind that fact to check anyone else's.

    alpha

  2. 02. A Best Lower Bound on Other Minds

    Absent proof, best answer I have found to the problem of other minds, of who is or is not a moral patient, is to use a similarity-weighted, pluralist prior.

    alpha

  3. 03. Competence Does Not Require Qualia

    The successionist error: treating capability as evidence of moral patienthood. It is not, and people are strangely reluctant to say so.

    alpha

  4. 04. The Universe With Nobody Home

    Not suffering, not human extinction as such: a cosmos in which nothing experiences anything, and which therefore has no moral relevance at all.

    alpha

  5. 05. Almost All Utility Functions Are Irrational

    Instrumental convergence, Bostrom and Yudkowsky, and an intuition about measure: the aligned targets are a vanishingly thin set.

    alpha

Part II · What To Do

  1. 06. The Hook

    A thousand-dollar littering fine, a fifty-dollar camera, and the small market that grows up around them.

    alpha

  2. 07. The Gist

    Hanson's fine-insured bounties in general form, and the conspiracy arithmetic that makes silence expensive.

    alpha

  3. 08. The Insurer

    The insurance market is the part that does the actual deterring, and it does it before anyone is ever charged.

    alpha

  4. 09. The Goal

    The target is not punishment and not even prevention of any particular project. It is to make the marginal researcher take a different job.

    alpha

  5. 10. Making It Global

    A mechanism that stops at a border relocates the risk instead of reducing it. What adoption would actually have to look like, and what it would still fail to reach.

    alpha

  6. 11. Compare and Contrast

    How this proposal stacks up against the other things people suggest, including on the axes where it loses badly.

    alpha

  7. 12. The Catches

    The strongest objections to the mechanism, collected in one place, stated at full strength, and answered only where I can answer them.

    alpha

  8. 13. Next Steps

    What a serious first version looks like, starting somewhere that is not AI at all.

    alpha


Recent writing

  • Response to Riley, 16 September 2026

    alpha 16 September 2026

  • Pacing Is a Setting on a Dial

    Anthropic's CEO now says frontier AI has to be slowed, and he has a three-step plan to do it. I agree with where he's going and disagree with the route in four places: what extra time is worth, who pays the evaluators, whether we should wait and see, and how much the flag on the lab matters.

    alpha 12 September 2026

  • When Your Insurer Collects the Bounty

    An insurer that turns in its own policyholder pays the fine and collects the bounty, netting roughly nothing. That arithmetic breaks the moment the policy excludes willful violations, which insurance law all but requires.

    alpha 11 September 2026