You’ve boarded a plane. The airline CEO walks down the aisle with a microphone.

“Ladies and gentlemen, we've done extensive testing on this aircraft. We believe there's a very good chance you'll arrive safely. There are some risks because we don’t fully understand aerospace science and there's a chance that any plane in flight, or possibly all of them, could fall out of the sky at any time and kill everyone aboard. We can’t estimate the odds and it’s out of our control, but our profits depend on having you make these trips, so sit back, relax, and enjoy your flight.”

You might not feel confident about your flight. Also you might run screaming for the door before it closes. If there’s a realistic possibility of catastrophe, why are they still selling tickets?

The CEOs of the giant AI companies are smiling and selling tickets to an array of AI products and telling us earnestly that AI could destroy society but their companies will be extremely valuable until the collapse.

There are three separate risks posed by AI, and we’re not dealing well with any of them.

(1) The development of AI is forcing a restructuring of society in unpredictable and possibly immensely destructive ways. 

(2) Some experts believe superintelligent AIs could escape human control. Sentient AIs are the villains in countless movies. Some experts and CEOs fear it’s not just science fiction.

(3) AI permits bad people to create cyberweapons vastly more effective than anything to date. An individual, a small group, or a state power could disrupt or collapse the global economy using hacks developed with the help of AI.

There’s something wrong with this timeline where the AI companies are proceeding with no restraints despite those risks.

Even a very tiny risk of existential harm to society justifies a huge response, ranging from the companies discontinuing their work to a rigid regulatory climate. 

By continuing their work the AI leaders bear moral responsibility for harm caused by AI as well as responsibility for public opposition to AI.

Grab a cup of tea and let’s follow a chain of reasoning that might seem depressing but it leads to hatred of billionaires, and that’s always fun, right?

Technology commentator Rob Reid has spent years thinking about catastrophic biological risks. (His TED talk will change your world.) He makes a powerful argument about biotechnology that applies equally to AI development. 

Imagine a new technology that carries only a one-in-100,000 chance of permanently ending civilization. Our instinct is to dismiss it because the risk is so small.

Reid argues that's exactly backwards. When the downside is the permanent collapse of civilization, even tiny probabilities deserve extraordinary attention because the consequences are almost unimaginably large.

Humanity has always contained a tiny number of unstable, malicious or reckless individuals. If new technology dramatically increases the damage a single person can cause, and the technology becomes cheap, decentralized, and widely available, yesterday's acceptable level of risk may become an existential threat. 

Reid uses the example of CRISPR gene editing tools, which were so expensive and difficult a few years ago that only well-funded labs could work with them. Today the price has plummeted and they have become far easier to use, making them available to a much wider audience.

AI models are getting smaller, smarter, cheaper, and increasingly capable of operating locally, placing powerful AI beyond the practical control of any single cloud provider. Anthropic is flirting with guardrails for Mythos, its most powerful model, by restricting it to a handful of companies for a few months but the restriction won’t last and other companies are not following suit.

Reid’s TED talk has a callback to the 2008 global financial crisis which looks amazingly prescient when you look at the behavior of today’s AI technology leaders.

“This needs to be an effort that spans society. We cannot ask a tiny group of experts to be responsible for both containing and exploiting this technology. We already tried that with the financial system, and our stewards became massively corrupted as they figured out how they could cut corners, inflict massive, massive risks on the rest of us and privatize the gains, becoming repulsively wealthy while they stuck us with the $22 trillion bill.”

You don’t have to decide for yourself if AI development is risky. The CEOs of the giant tech companies are the ones warning about human extinction, biological weapons, loss of control and civilization-ending risks.

In May 2023 more than 600 high-level AI experts and tech experts signed a statement declaring that mitigation of the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. Signatories included the CEOs and scientific architects from OpenAI, Google DeepMind, and Anthropic.

A few months before co-founding OpenAI in 2015, Sam Altman published a thoughtful essay warning that the development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.  In January 2023, Altman stated that the bad case for AI was "lights out for all of us.”

Dario Amodei, the CEO of Anthropic, has consistently warned that AI models will be capable of filling in the missing technical steps required to execute large-scale biological attacks. 

Amodei has also suggested that AI could eliminate as much as half of all entry-level white-collar jobs within five years. 

Paul Christiano, the founder of the Alignment Research Center (ARC), and Mustafa Suleyman, CEO of Microsoft AI, worry about a gradual pathway to civilizational collapse from the systematic enfeeblement of human society. 

Suleyman recently said that another central worry is “that many people will start to believe in the illusion of AIs as conscious entities so strongly that they’ll soon advocate for AI rights, model welfare and even AI citizenship. This development will be a dangerous turn in AI progress and deserves our immediate attention."

Last week Eric Schmidt, former Google CEO, wrote this in the New York Times:

“Too little caution (in managing AI development) could unleash labor-market chaos and social disorder. Balance matters. If we get the balance between control and growth wrong at any point, we’ll fall into the gorge below. And no country is guaranteed to get to the opposite side.”

I am focusing here on the statements of CEOs and tech company leaders because they bear the responsibility for continuing their work at breakneck speed, and for capturing the regulators (basically the entire US government) to ensure there is no regulation whatsoever to address the risks.

If you want to evaluate the risk fully, though, cast a wider lens and look at the alarms from scientists, researchers, and AI experts going back many years. Last week nearly 200 people, including 15 Nobel laureates and the chief economists of Open AI and Anthropic issued yet another public statement warning of “risks, including large-scale job displacement.”

Money and power. It’s always money and power with these guys.

Let’s be maximally cynical because I think they’ve earned our distrust and cynicism. 

Describing AI as an existential risk is a powerful marketing tool. It makes AI seem important and unstoppable. The CEOs issue the apocalyptic warnings. The politicians use the warnings as an excuse to permit the industry to move forward at full speed, claiming that we have to rush out of fear that other countries might gain an advantage.

And the predictions of massive job market disruption are like red capes waved in front of all the businesses looking for advantages and worried about being left behind.

The result is that the AI companies become extremely valuable.

The burden should be on the CEO leaders to explain why the benefits of AI outweigh the extraordinary risks they themselves describe. But I’m suggesting that it doesn’t really matter. There are only two possibilities.

One is that the dire warnings are exaggerated to juice the companies’ stock prices. Then the industry’s own rhetoric has helped manufacture public anxiety. People are making career choices and dealing with emotional problems and crippling feelings of powerlessness because of the AI hype.

The other is that the leaders genuinely believe what they say. In that case, they are knowingly building a technology with at least some possibility, however remote, of producing irreversible catastrophe. 

Either way, the CEOs and AI architects bear the moral burden of their actions - not governments, not customers, not investors. The responsibility for the destabilization of society and the fracturing of public trust rests with the people making the decision to continue.

History will not judge them by the eloquence of their warnings or by their personal fortunes. They will be judged by the catastrophes they failed to prevent.