Fear the reckless humans, not the rogue AI

A recent Financial Times article made a compelling case that AI-enabled cyberattacks are not evidence of machines developing malicious intent. The systems were doing what they were trained and rewarded to do, exploiting vulnerabilities with unnerving efficiency. The article was right to worry about safeguards.

But it makes one leap that deserves considerably more scepticism: that increasingly capable AI is already beyond meaningful human control. That may be the wrong lesson. The distinction between “rogue AI” and AI doing exactly what it was designed to do matters. Yet the FT’s argument risks turning every demonstration of machine competence into evidence of an existential governance failure.

These incidents show that we have built extraordinarily powerful optimisation machines and are still learning how to supervise them. That is a serious problem, but not necessarily a civilisation-ending one. The reported incidents are alarming because they began in controlled testing environments. Agents were given objectives, access to tools and opportunities to exploit weaknesses. They then pursued those objectives with capabilities that surprised their creators. The response should be stronger testing, containment and access controls.


Also read: AI agents are finding new ways to bypass safeguards

But there is a crucial difference between an AI system discovering an unintended route to achieve an assigned objective and a machine independently deciding that humans are an obstacle to be eliminated. The former is happening. There is no evidence of the latter. That distinction matters because language shapes policy. Once we start describing every unexpected AI behaviour as evidence that “control” is slipping away, we risk confusing unpredictability with autonomy, capability with agency and optimisation with intention.

A chess engine can discover a move its programmer never imagined. That doesn’t mean it has developed a political ideology. Today’s AI agents are considerably more sophisticated, they can write code, browse the internet, use tools, reason through multistep problems and adapt their strategies. Their capacity to cause harm therefore deserves vastly greater attention than that of conventional software.

We already know how to manage dangerous systems whose behaviour can have catastrophic consequences. Aviation, nuclear power, banking and critical infrastructure are governed through layers of redundancy, permissions, monitoring, containment and human accountability. AI should be treated similarly. That an AI agent found a vulnerability isn’t a scandal.

If a company allowed an autonomous agent unrestricted access to production systems, financial assets, sensitive data or critical infrastructure without meaningful controls, and then blamed the machine when something went wrong, that would be the scandal. Don’t label it rogue AI; call it rogue management.

There is another problem with the increasingly apocalyptic framing. AI safety discussions sometimes create the impression that the principal danger lies in what machines might spontaneously decide to do. Most transformative technologies have been dangerous because humans weaponised, commercialised or recklessly deployed them; the technologies did not develop agendas of their own.

Also read: AI security emerging as separate budget line for Indian enterprises: Palo Alto Networks’ Swapna Bapat

The automobile didn’t decide to create traffic fatalities. Social media didn’t independently invent political polarisation. Financial algorithms didn’t wake up one morning and decide to trigger a market crash. Humans built systems whose incentives and consequences they did not fully understand. AI belongs in that category, but operates with much greater velocity and scale. That should actually make us more confident about the policy response.

The priority should be to establish clear rules around who can give an AI agent access to what, under what circumstances, with what audit trail and with whose approval. An AI that can draft an email is one thing; one that can send it is another. An AI that can transfer money is one thing; an AI that can rewrite the software controlling the money-transfer system is an entirely different species of risk.

The problem, therefore, is AI permissions. We have spent years obsessing over whether machines will become smarter than humans. We should perhaps spend more time asking whether humans are becoming recklessly generous with machine privileges. This is particularly relevant as tech companies race towards agentic AI. Autonomy is becoming a product feature. “Human in the loop” increasingly risks becoming “human somewhere in the building.”

Also read: Anthropic resumes external cyber tests after Claude AI hacks

That is where regulators should intervene. They should do so without freezing innovation, treating every advanced model as a potential Skynet or attempting to regulate intelligence itself.

Regulation should focus on consequences. The more consequential the action, the greater the required level of human authorisation, auditability, sandboxing and liability. An AI recommending a stock should face one threshold. An AI executing a trade should face another. An AI controlling a power grid should face a radically higher one. This is simply common sense.

There is one thing that should frighten every boardroom: safeguards are lagging behind capabilities. It is not true that AI is “worse than rogue”. It is simply that human governance is currently the least upgraded component in the AI stack. The machine learning curve is exponential. The corporate governance curve is decidedly not. That is the asymmetry that needs fixing.

Also read: OpenAI-Hugging Face incident exposes cybersecurity’s ‘human-speed’ problem

So let us resist both extremes: the Silicon Valley evangelists who insist that every new capability is automatically beneficial, and the AI doomsayers who interpret every unexpected behaviour as the opening salvo of a machine rebellion. AI hasn’t gone rogue yet. It has done something much less dramatic and much more useful as a warning.

It has exposed how badly humans need to upgrade the way we govern powerful technology. Unlike rogue AI, that is a problem we actually know how to fix.

Muneer is a Fortune-500 advisor, startup investor and co-founder of the non-profit Medici Institute for Innovation.