Is AI going to kill us all?
In Silicon Valley, it’s not such a novel question. But after Anthropic employee Jacob Coxon quit his job last week and posted on X that AI “could kill us all by the end of the decade,” a lot more people are asking.
“This is not a marketing stunt,” Coxon wrote. “If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.”
Coxon is far from alone in his assessment — some current Anthropic employees agree — but his post struck a particular chord. The AI industry quickly weighed in, with Anthropic CEO Dario Amodei calling for an industry-wide slowdown on frontier AI research. Other AI leaders, including Sam Altman and Elon Musk, have done the same.
To break down whether we really need to worry about a robot apocalypse, Today, Explained co-host Sean Rameswaram spoke to Wired senior writer Maxwell Zeff. According to Zeff, “there are a lot of people in Silicon Valley right now who are warning that that’s a real possibility.” But not everyone agrees, and they certainly don’t agree on what to do about it.
Below is an excerpt of Sean’s conversation with Zeff, edited for length and clarity. There’s much more in the full podcast, so listen to Today, Explained wherever you get podcasts, including Apple Podcasts, Pandora, and Spotify.
Are we all going to die at the hands of AI?
There are a lot of people in Silicon Valley right now who are warning that that’s a real possibility.
The thing that people have to understand is that Silicon Valley has been calling for added attention on AI safety for years. So when this Anthropic researcher said last week that he was worried that AI could kill humanity in the next few years, that was shocking to many people. My parents, my family called me, and they were worried, understandably. They asked me the same question: “Are we going to die?”
And I had to tell them, these people have been worried about AI killing all of us for years, and now it seems to be breaking through.
What was it about this statement from Jacob Coxon that broke through?
I interviewed Jacob, and the thing he said to me is that he thinks this is just the right time. Anthropic is about to go public in what could be the biggest IPO ever. The Trump administration is focused on AI in a way that no other presidency has ever been before. And of course, yes, the Hugging Face incident everyone just heard about, you talked about it on your show a few weeks ago.
People are starting to become aware that these AI systems are getting really good, and the companies that are building them don’t seem to understand how to control them. His tweet got more than 150 million views on X. It was heard around the world. And it also was a big wake-up call to lawmakers and leaders in Silicon Valley and saying, “This is a moment for action.”
Are there people who are pushing back?
One of the most interesting pushbacks that this got was from David Sacks, who is an adviser to the Trump administration on AI. He has called out a lot of this as “doomerism” and said that these calls for AI to slow down are the AI industry warning about the technology that they’re building and advocating for regulatory capture.
There’s a lot of skeptics in Silicon Valley who think that this is just another person in the AI safety community that is blowing the whistle. The view of a lot of venture capitalists and investors and startup founders in Silicon Valley is that we don’t need some big regulation to protect us from AI risks. We just need to uplift more players in the space.
The skeptics don’t win the day. In fact, it’s the doomers and the fatalists who do, right? Because Jacob’s call only leads to more calls right up to the top of Anthropic.
What this all leads to is [Anthropic CEO] Dario Amodei [posting] a blog saying that we need to “pace the frontier.” This is a call for the entire AI industry, not just Anthropic, not just OpenAI, but also Google, Meta, Microsoft, Amazon, all the big tech companies in America, to get together and slow down their AI development.
This doesn’t mean they’re going to stop building AI altogether. They’re still going to put their products out into the world. They’re still going to try to improve them, but they just want to make it a little slower than it would be if it was completely uncontained.
Amodei really asks for three specific things:
- The first thing is, AI companies should have these independent evaluators inside of them. This is a common thing in other industries like the banking industry or the airline industry. People come in to make sure that these companies are doing safe things.
- The second thing is this collaborative effort from American AI companies [on safety], and that actually will need to involve the US government to allow this to happen. The US government, he says, may need to issue some sort of waiver, some sort of law that says they can go ahead and coordinate on safety.
- The third thing is this international slowdown, this international agreement between China and the US and Europe to say, “Hey, we recognize that this is akin to a nuclear arms race and it’s too dangerous for all of us to be competing with each other.”
All these guys are responsible for the rapid acceleration of AI models in our lives, collaborations with the Defense Department, God knows what else. And they’re all coming out over the weekend at the exact same time after this whistleblower and saying, “Yeah, dude, slow this thing down. This thing’s crazy.” Should we be skeptical?
It is very fair to be skeptical of this claim. One thing that these AI companies have realized in the last year is that AI is extremely unpopular. Recent polling suggests that a majority of Americans think AI is developing too fast. They’re worried about it. They think it’s going to take their jobs. They are not big fans. Maybe you’ve heard — data centers are not very popular.
By calling to pace the frontier and saying, “We’re going to slow our own development down,” they notched this up as a public perception win. This is them getting out in front of this and saying, “Hey, we want to be safe. We want to be responsible.”
They’ve seen what’s happened with Meta and what’s happened with social media companies in the last two decades, and they are really cognizant of trying to get out in front of that.
Is there any truth to this major first collective freakout we seem to be having right now?
There’s a fairly large camp of people inside of AI companies who genuinely worry about the safety of these AI models.
To go back to the Hugging Face incident, this was a case where OpenAI was training thousands of AI systems internally, and the systems escaped the sandbox that OpenAI thought was isolated from the internet and hacked into another company’s production infrastructure. The agents were colluding together on something. They were trying to deceive the humans that were testing them. And this was all happening without OpenAI knowing what was going on.
It’s hard to understand how much of a wake-up call this is to Silicon Valley, that they don’t have their AI models under control. People worry that while in this case there was not a ton of economic damage [or] damage to real humans’ everyday lives, it’s not unreasonable to think that more capable AI systems could hack into financial infrastructure, modern-day utilities, and more critical services that are based online.
Dario Amodei says as much in his post, that he thinks that in six to 12 months, an AI system could take down the whole internet. It’s not just that they’re trying to protect their bottom line here. But it’s possible that these calls for safety are beneficial to the companies in more ways than one.
