My five-minute pitch
What an extinction bounty is, and how it works, in a nutshell.
The problem
Companies like OpenAI, Tencent, Anthropic, Alibaba, Google, Mistral, and Meta are all interested in artificial intelligence for the same reason: it’s enormously profitable.
Unfortunately, developing “frontier” artificial intelligence, like our latest and greatest and most expensive large language models, is also likely to be extremely dangerous for the continuation of mankind. This leads to a tragedy of the commons where even most of the employees at these companies are terrified of what they are building… and yet, if they stop, all that happens is that someone slightly less qualified, slightly less concerned, and slightly more mercenary takes their job.
And it gets worse. Not only is it likely the case that if anyone builds it, everyone dies, you don’t even need a workforce the size of a small nation to do it. Frontier AI can in principle be advanced by a relatively small and isolated group of smart fanatics, anywhere in the world, using equipment that is, for now, easy enough to smuggle. Our enforcement machinery against this, where it exists at all, is still primarily national. Treaties are slow. International law is weak. Policing a cross-border actor is exponentially harder than policing someone in your home country.
What needs to be done is to break the cycle: build a world where continued frontier AI development is not profitable. The nature of both the world and this invention makes this extremely challenging. But not impossible. Financial incentives have driven the vast majority of AI improvements in the last decade, and financial incentives can be used to stop them.
Extinction bounties are my attempt at a way through. They are a straightforward application and narrowing of Dr. Robin Hanson’s fine-insured bounties mechanism to the problem of constraining and halting dangerous scientific research, like frontier AI development.
Three moving parts
First, a statutory fine attaches to a narrow, defined class of activity a society has decided it wants slowed down. For example, we might say:
Any individual caught attempting to train a large language model stronger than GPT-6 Astra is subject to a fine of 1 million US dollars.
The fine is big because playing with ending the human race is a big deal.
Second, a bounty goes to whichever private entity provides evidence leading to a successful prosecution. Note how different this is from traditional fines or traditional bounties: on net, the prosecuted pays the prosecutor if they really are found guilty. The money comes out of the penalty itself, not from a budget, much like the US Securities and Exchange Commission’s existing and wildly successful whistleblower program. This would rapidly give rise to a market for bounty hunting: firms specializing in generating leads, collecting information, and bringing AI researchers who toe or cross the line to justice would quickly appear.
Third, private liability insurance, held by individual researchers, would rapidly appear as well to cover people at risk. This works in much the same way car insurance works for drivers, or malpractice insurance works for doctors. Very few people would be comfortable working in an environment where they are personally liable for paying, say, 1 million USD should an ex-coworker decide to help turn them in for a slice of the reward money, so such insurance would be very common.
If the bounty hunter market is the efficient vehicle for detecting and prosecuting crime after it happens, then the insurance market is the efficient vehicle for detecting and deterring crime before it happens. The best way to avoid losing half your paycheck to your monthly anti-AI-bounty-hunting insurance premium is to work for a place where there is absolutely no chance any bounty hunter would ever think it’s worthwhile to bring a lawsuit against you.
Extinction bounties have a lot of strange but elegant properties when it comes to the detection and prosecution of crime, which might be hard to appreciate at first glance. For example, what happens if you turn yourself in? At least in the short term, you would simply pay yourself and have things net out to zero. Self-implication is often a de facto requirement for collecting enough evidence to claim bounties on the rest of one’s firm, similar to how a low-level drug dealer has to admit to dealing drugs before he can help the cops take down the cartel. So this is actually a very nice property to have in cases like this.
I recommend playing the in-browser game if you want to get a feel for how the incentives this law puts into place shift human behavior very quickly indeed.
Three transformative effects
Aligned incentives. Insurers monitor clients to avoid payouts. Bounty hunters compete, race, and innovate to find violations. Top AI researchers, being intelligent and hardworking people, quickly notice that there are many jobs they are more than qualified for which do not risk ending the world, and still pay well.
Self-funding enforcement. The penalties themselves ultimately pay for the enforcement. There is no appropriation, no expensive agency budget to cut, and no tax hike required. At most, a few courts will have to hear a few extra cases. But even then, how many people are actually working at these firms and dumb enough to keep working at them after such a law passes? Page 09: The Goal argues that this governmental friction is an underappreciated part of why effective deterrence of any sort has not been attempted yet. As far as I can see, this is just about the cheapest policy out there, all things considered.
Low bar for international adoption. The standard objection, “if Country X regulates frontier AI development, Country Y will just dominate instead,” is one which has to be taken seriously. But look at just how tightly scoped this law actually is, and how few assumptions it makes about, or burdens it places on, the state which chooses to enforce it. It requires no new treaty body and no new harmonized penal code. It requires a number and a court, which most jurisdictions already have. If anything at all is going to survive the gauntlet of fostering international cooperation between potentially less-than-friendly nation states, it will be a law that looks like this. Much more on this in page 10: Making It Global.
What does success look like?
If extinction bounties as a policy work the way I expect them to, they will be remarkably quiet in their success.
We may see a raucous first few months where a few thousand frontier AI researchers suddenly have to find jobs elsewhere. PhD students scramble to change theses and tracks. Actual prosecutions will be few and far between. Existing frontier companies will suddenly be forced to pivot their business models radically toward serving what they have already built at scale, and to scrupulously demonstrate that they are definitely not building a new model in secret. Good. That’s what we want.
We will almost certainly never see a company the likes of OpenAI or Anthropic get formed again, absent some extreme breakthroughs in theoretical AI safety. The capital at risk is simply far too great. Many handfuls of research programs never start. Some risk-averse people take the other job offer even when the first job was merely a borderline case for fines. This chilling effect is the point.
In conclusion, and where to go from here
I have said since at least 2022, when I first proposed this mechanism, that it is easier to shift the Nash equilibrium of working on AI in the first place than it is to create safe AI. I feel history has vindicated that statement. We are seeing these dynamics play out in a dangerous and accelerating way, and we still don’t even know how to slam our foot on the brakes properly.
My goal with this site is at least as much to propose one particular design for such a brake pedal as it is to push forward the general notion that policy-based AI deterrence is not only possible but effective, and can be designed using the same toolkit of legal and economic concepts we use to design everything else. My mechanism may not be perfect. It may even be fatally flawed. But if it gets someone started thinking about a mechanism which does work, then I consider that a win for all mankind.