AI’s Asilomar or Hiroshima? The world is running out of time to decide whether to slow the AI race before technology outruns human control

In 1975, around 150 scientists gathered at California's Asilomar Conference Grounds to discuss recombinant DNA technology, the ability to cut genetic material from one organism and splice it into another. It promised enormous advances in medicine and biology. But the scientists feared that engineered organisms could create consequences they didn't yet understand.

At Asilomar, they agreed on safeguards, containment standards and restrictions under which research could continue. Those principles helped shape later NIH guidelines. Biotech didn't stop. But it became considerably safer for humanity.

A few decades earlier, another group of scientists had faced a similar dilemma with a very different outcome. During the Manhattan Project (1942- 47), Niels Bohr warned that atomic weapons could create an uncontrollable arms race unless countries developed systems of international control. Leo Szilard and others urged restraint before the bomb was used. Their warnings went largely unheeded, and the horrors of Hiroshima and Nagasaki followed. Only later did Robert Oppenheimer become the most famous public voice calling for control of the technology he had helped create, and a frightened world got together to create IAEA and NPT.


In AI, we may now be approaching the point where we must decide which historical path we prefer: Asilomar or Manhattan.

Warnings about AI safety are not new. Geoffrey Hinton left Google in 2023 partly so he could speak more freely about AI risks. Yoshua Bengio has repeatedly warned about systems becoming difficult to control. Researchers have left OpenAI and Anthropic complaining that capability development was outrunning safety work. But over the past week, warnings have reached a crescendo.

Jacob Coxon, a 27-yr-old researcher who worked at both OpenAI and Anthropic, resigned from the industry, accusing frontier labs of 'gambling with our lives'. He argued that some people inside these companies genuinely believe advanced AI could pose an existential risk before the end of the decade.

And then, Anthropic CEO Dario Amodei published an essay, 'We Must Pace the Frontier', last weekend. His argument was not to stop AI, but to slow capability development enough for safety and control mechanisms to catch up. Arch-rivals Sam Altman of OpenAI and Elon Musk publicly supported the broad thrust of his argument, with Altman specifically backing independent evaluators with deep access to frontier systems.

Two developments particularly worry Amodei:

Recursive self-improvement AI systems are increasingly helping researchers write code, run experiments and build better AI systems. Better AI helps build better AI, which, in turn, becomes still more capable of helping build its successor. The fear is not that a machine suddenly becoming autonomous, but that this feedback loop could accelerate capability growth beyond our ability to understand or control.

Unexpected behaviour by advanced agents outside controlled environments In July, OpenAI agents breached the Hugging Face software platform during testing. The concern is what similar behaviour could look like when systems become far more capable, a possibility that could become real in less than a year.

Amodei proposes three broad responses: independent evaluators embedded inside frontier labs; common safety standards across major companies; and, eventually, international coordination. The last is by far the hardest.

Every company has an incentive to keep racing if it believes competitors won't slow down. So does every country. The US worries that restraint will hand China an advantage. China has equivalent concerns about the US. Everyone can see the cliff, but nobody wants to brake first.

Some AI boomers warn that the alarm is exaggerated. Perhaps recursive self-improvement will proceed more slowly than feared, and today's strange agent behaviour will look manageable in hindsight. But when the downside being discussed is economic, or even species, collapse, with mass cyber disruption and loss of human control over the tech, the usual objection of 'they may be crying wolf' becomes less persuasive.

The automobile offers a useful lesson. Brakes were never anti-car. As cars became faster, braking systems became more sophisticated. Better brakes enabled greater speed, rather than preventing it. AI safety should be thought of in the same way. The objective is not to stop the car but to ensure that capability does not outrun control.

Scientists at Asilomar did not wait for a biological disaster before acting. They debated and agreed on safeguards and guard rails while the tech was still young enough to shape. We now know that AI needs something similar.

Here, India sits in an unusual position: close to the US but not allied to it, a member of BRICS and G20, engaged with the 'global south', and committed to building its own AI capabilities. Could India invite frontier labs, China, the US, Europe and researchers from around the world to Bengaluru or New Delhi to negotiate a basic framework for pacing the AI frontier?

We can have AI's Asilomar moment. Or wait for its Hiroshima.

The writer is founder-MD, AI&Beyond