AI deployment needs strong checks, mate!

As enterprises race ahead with AI adoption and the state intensifies its AI-led governance push, the main question is no longer whether AI should be deployed, but whether the guardrails are strong enough to prevent systemic harm. With banks, corporations and public services relying heavily on algorithms, experts warn that blind trust in AI could lead to distorted judgment, expose vulnerabilities and create significant risks in the absence of clear oversight.

Take the case of cybersecurity. The real risk in this area is analysts becoming overly dependent on AI tools, said Huzefa Motiwala, senior director, Technical Solutions, India and Saarc, Palo Alto Networks.

“Suppose we accept machine output at face value,” he said. “In that case, we miss the bigger picture, and that’s where mistakes happen.”

Criminals are probing weaknesses through prompt injection, data manipulation and flooding systems with noise. According to Motiwala, it’s essential to test AI systems in challenging situations, conduct mock attacks to identify weaknesses, and not underestimate the human-in-the-loop model. “If outputs are treated as infallible, we risk wrongly freezing accounts or excluding beneficiaries at scale. The safer path is transparency, audits for bias and blind spots, and a clear appeals process,” he said. The vulnerabilities begin with data itself, said Anushree Verma, senior director analyst, Gartner.

“It is dangerous for enterprises adopting AI at scale if not implemented with the right measures and guardrails,” she said. “If the data is biased, not engineered, and prevented from bad actors, it is ‘garbage in, garbage out’.”

She emphasised that operating models, clear policies and controls, and enabling oversight technologies are the three pillars upon which effective AI governance must be built. Furthermore, she highlighted the increasing importance of AI TRiSM (trust, risk, and security management) methods, which offer enterprises insight into the application of AI and ensure decisions are in accordance with their corporate objectives and legal demands.

However, without regulatory backing, organisational governance cannot do much.

Subimal Bhattacharjee, policy advisor to enterprises and government, flagged broader risks.

“The key risks include algorithmic bias affecting citizen access to services, lack of transparency in automated decision-making, potential for AI systems to perpetuate inequalities, and cybersecurity vulnerabilities in critical infrastructure,” he said.

India currently has no AI-specific law and relies on the provisions of the IT Amendment Act 2008. India currently lacks an estimated implementation plan for comprehensive AI regulation. As AI adoption picks up, this is resulting in regulatory gaps. India should act swiftly to establish accountability and transparency standards, mandate algorithmic audits for government AI platforms, and demand human oversight for important decisions, Bhattacharjee said.

Experts argue that India’s AI moment is not just about a technological leap and advancement but a governance test as well. The main challenge is making sure that AI tools continue to be efficient and effective assistants rather than unsupervised decision-makers.