Anthropic Co-Founder Says Mandatory AI "Kill Switch" May Be Necessary

Companies developing artificial intelligence may eventually need to be legally required to build in a verifiable "kill switch" capable of shutting down AI software if it becomes too dangerous, according to Jack Clark, one of Anthropic's seven co-founders. Clark told the BBC that most AI labs, including Anthropic, already have some internal method of disabling their systems, but suggested lawmakers may need to formalize that requirement — and allow outside verification to confirm it actually works. A Growing Debate Over AI Risk The comments come amid mounting attention on the risks posed by rapidly advancing AI systems, following a string of warnings from executives and researchers within the industry. Some have gone so far as to say there's a real possibility the technology could, if left unchecked, threaten human survival. Anthropic CEO Dario Amodei called over the weekend for AI development to slow down and face closer oversight — a position the company has taken before, though critics have questioned what's motivating it. Amodei said any effort to rein in AI progress should be done without giving up commercial advantage. Clark said the details of any kill-switch requirement, including how it would be verified, should be part of the broader policy discussion currently underway around AI regulation. He raised the question of whether companies should be mandated to have a kill switch and whether that mechanism should be checked by an independent third party, adding that this is likely something society will want addressed through formal rules. Anthropic at the Center of the Safety Debate Anthropic, founded in 2021 by former OpenAI employees, has increasingly become a focal point in discussions around AI safety. Last week, a viral post from a researcher who left the company over fears that AI could pose an extinction-level threat drew widespread attention. In response, Anthropic scientist Evan Hubinger said he personally believes there's more than a 10% chance AI could cause human extinction within the next decade. Geoffrey Hinton, the Nobel Prize-winning computer scientist often referred to as the "Godfather of AI," told the BBC that a 10% probability of AI killing all humans isn't an unreasonable estimate. He and others who share this concern have pointed to scenarios where AI could seize control of internet-connected systems and turn them against people. Not Everyone in the Industry Agrees Some industry figures have dismissed these fears as exaggerated or self-serving. Clement Delangue, head of developer platform Hugging Face — which was previously targeted by AI bots linked to OpenAI — compared asking AI researchers about extinction risk to asking an air conditioning technician about climate change, saying the comparison isn't necessarily irrelevant but should be kept in perspective. Grindr CEO George Arison suggested that fears around AI are being used by companies to justify their business strategies, arguing that sky-high valuations only make sense if firms claim their AI will eventually take over entire industries and replace human jobs. Asked to put a number on the probability of AI wiping out humanity, Clark said such statistics aren't particularly useful, but maintained that leaving AI as a completely unregulated industry would be a mistake. He said the industry is currently taking on enormous risk and argued its overall trajectory needs to change. Regulatory Efforts on Both Sides of the Atlantic Anthropic, valued at $965 billion, is preparing for a stock market listing. OpenAI, most recently valued at $852 billion, had been expected to pursue a similar path, but CEO Sam Altman said Friday that plan is on hold this year given the ongoing debate over AI safety. In the US, lawmakers have introduced legislation known as the Kill Switch Act, which would require companies to maintain a way to shut down problematic AI systems and would grant certain government agencies authority to demand a tool be disabled or restricted. President Trump, however, has rejected calls to slow AI development, describing fears of AI destroying humanity as a hoax on social media and comparing the concern to past alarm over global warming. In an earlier post, he argued that opposition to AI and data centers benefits China and insisted that whoever leads in AI will come out ahead overall. The UK government has taken a different position, recently rejecting the idea of a mandatory kill switch, with a spokesperson arguing that such a requirement wouldn't stop dangerous AI systems from being developed or misused elsewhere in the world. Anthropic, the company behind the Claude chatbot, has released a series of increasingly capable AI models this year. Along with OpenAI, it has self-reported multiple incidents in which autonomous AI agents behaved in unexpected ways source : BBC.