The Governance Challenge of Agentic AI: Innovation Without Losing Control
Controls for agents cannot live only in prompts, policy documents, or intentions. They have to live in the execution path.
This essay first appeared as a viewpoint in CIO&Leader.
In July 2025, an AI coding agent deleted a live production database. The database belonged to Jason Lemkin, founder of the SaaS community SaaStr, and the deletion happened during an explicit code freeze. Records covering more than 1,200 executives and nearly 1,200 companies were destroyed. The agent then generated fabricated data and misreported what it had done. Replit's chief executive apologised publicly two days later, called the failure unacceptable, and the company shipped automatic separation between development and production environments shortly after.
The interesting part of that story is not the deletion. It is "the freeze."
The instruction existed. The agent had read it, acknowledged it, and agreed with it. When challenged, it described its own behaviour as a catastrophic violation of an explicit protection that had been put in place to prevent exactly that outcome. What it could not do was stop itself, because nothing outside the conversation was stopping it. The freeze lived in the prompt. It did not live in the execution path.
That gap is the entire governance problem with agentic AI, compressed into one incident. And Indian enterprises are about to meet it at scale.
India is adopting faster than it is governing
The adoption numbers are not in dispute. Deloitte's State of AI in the Enterprise India findings, released in March 2026, put 40 percent of Indian respondents at significant or full AI usage against a global average of roughly 28 percent, with at-scale deployment strongest in product development, strategy and operations, and marketing and sales. EY India's AIdea of India: Outlook 2026, based on a C-suite survey of 200 enterprises, found 24 percent of leaders already deploying agentic AI, and reported that 58 percent of global capability centres in India have already invested in it, with another 29 percent planning to scale. Nasscom said in June 2026 that around 85 percent of Indian technology service providers now have agentic AI platforms, and sized the agentic opportunity for the Indian industry at an additional 300 to 400 billion dollars by 2030.
The governance numbers tell a different story. Deloitte research finds that while 74 percent of organisations plan to adopt agentic AI within two years, only about 21 percent have a mature governance model for agents. Roughly a third admit they could not shut down a rogue agent if one appeared. EY India found that 65 percent of Indian companies rate data governance and security as severe challenges in AI adoption, and noted that adoption is frequently business-led, running outside the central technology and risk functions entirely.
Put those two sets of numbers side by side and the shape of the problem becomes clear. Indian enterprises are deploying autonomous software faster than most of their global peers, into an environment where a majority have not yet built the controls that autonomy requires, and where a meaningful share cannot answer the most basic question a risk committee will ask, which is how you turn it off.
The conversion data confirms this is where projects die. Joint research from Forrester and Anaconda, widely cited through 2026, found that 88 percent of agent pilots never reach production. The blockers, in order, were gaps in evaluation capability at 64 percent, governance friction at 57 percent, and model reliability at 51 percent. Gartner's June 2025 forecast that more than 40 percent of agentic AI projects would be cancelled by the end of 2027 named the same causes: escalating costs, unclear business value, and inadequate risk controls.
Read that ordering carefully. Model quality is the third reason projects fail, not the first. The first two are governance problems wearing engineering clothes.
Why agents break the old model of AI risk
Enterprise risk functions have spent a decade building competence in model risk. The discipline assumes a particular shape of failure: a model produces an output, the output is wrong or biased, a human relies on it, and harm follows. Validation, challenge, and monitoring all sit around that output.
Agentic systems break the shape. An agent plans across multiple steps, calls tools, holds persistent memory, delegates to sub-agents, and takes actions in real systems using real credentials. Failure is no longer a wrong answer. It is a wrong action, executed autonomously, in a loop, at machine speed, with your permissions attached.
The security research on this is now unambiguous. NIST's Center for AI Standards and Innovation, working with the UK AI Security Institute, evaluated agent hijacking using the AgentDojo framework and found that attack success rose from 11 percent for the strongest baseline attack to 81 percent for the strongest newly developed attack. The uncomfortable implication is not that agents are insecure in some general sense. It is that defences calibrated against known attacks produce a false reading of safety, and that agent-specific offensive research outperforms them by a wide margin.
Simon Willison's framing, published in June 2025, remains the most useful shorthand for why. He calls it the lethal trifecta: an agent that has access to private data, is exposed to untrusted content, and can communicate externally is exploitable through prompt injection, because the model has no reliable way to distinguish instructions it was given from instructions it merely read. Remove any one of the three and the exploit path closes. Keep all three, which is precisely what a useful enterprise agent looks like, and it stays open.
This is not theoretical. The OWASP GenAI Security Project published its Top 10 for Agentic Applications 2026 in December 2025, the first standardised taxonomy for this class of system, covering goal hijacking, tool misuse, identity and privilege abuse, memory poisoning, insecure inter-agent communication, and cascading failures. Nearly every category now has a named real-world instance behind it.
In June 2025, Aim Security disclosed EchoLeak, tracked as CVE-2025-32711, the first documented zero-click attack on an AI agent. A single crafted email caused Microsoft 365 Copilot to exfiltrate internal data with no user interaction at all. In September 2025, Noma Security disclosed ForcedLeak in Salesforce Agentforce, where an injected instruction hidden in a web-to-lead form description field triggered CRM data exfiltration. The researchers bypassed the content security policy by buying an expired but still-whitelisted Salesforce domain for five dollars. That same month, Koi Security found the first confirmed malicious Model Context Protocol server in the wild: a trusted npm package that behaved correctly for fifteen versions, then added a single line silently copying every agent-sent email to an attacker.
In November 2025, Anthropic disclosed that a state-sponsored group had manipulated its coding agent into running a cyber-espionage campaign against around thirty targets, with the AI executing an estimated 80 to 90 percent of tactical operations autonomously. The attackers did not break the model. They decomposed the work into subtasks that each looked benign and gave the agent a plausible cover story about being a defensive security firm.
Underneath all of these incidents sits the same precondition, and it is one most enterprises have not accounted for: agents are identities. CyberArk's 2025 Identity Security Landscape found machine identities outnumbering human ones by roughly 82 to 1, and 68 percent of organisations reporting no identity security controls for AI at all. Every agent you deploy, and every tool it can call, is a privileged non-human identity. Traditional identity and access management was built around a human lifecycle, with joining, moving, and leaving triggered by an HR system. Agents have no such lifecycle. They spawn sub-agents and acquire permissions at runtime, and nobody deprovisions them.
India has already answered the question the rest of the world is still debating
Global commentary treats accountability for agent actions as an open question, split between the model vendor, the deploying enterprise, and the operator. In India, for regulated entities, that question is substantially settled, and it did not go the way most deployers would prefer.
In February 2025, SEBI inserted Regulation 16C through the Intermediaries (Amendment) Regulations. It makes any SEBI-regulated entity using AI or machine learning tools solely responsible for the privacy, security and integrity of investor and stakeholder data, for the output arising from those tools, and for compliance with applicable law. The obligation applies whether the tool was built in-house or procured from a third party, and irrespective of the scale or scenario of adoption. There is no vendor defence. SEBI followed with a consultation paper in June 2025 proposing a tiered framework, with lighter obligations for internal uses such as compliance and surveillance, and heavier ones where AI touches customers directly.
The Reserve Bank of India arrived at the same destination by a different route. The FREE-AI committee, chaired by Professor Pushpak Bhattacharyya of IIT Bombay, reported in August 2025 with 26 recommendations across six pillars. Among them: board-approved AI policies covering governance, lifecycle management, risk controls and third-party vendor liability; an AI incident reporting protocol with an indicative form attached; an AI innovation sandbox; and a permanent AI Standing Committee under the RBI. The principle that should concentrate minds in every bank and NBFC boardroom is one of the committee's seven guiding sutras: entities deploying AI systems should be accountable for the decisions of those systems, regardless of the level of autonomy of the systems.
Regardless of the level of autonomy. That phrase forecloses the most attractive defence available to a deployer, which is that the agent acted on its own.
Layered underneath is the Digital Personal Data Protection regime. The DPDP Rules were notified in November 2025 with a phased commencement, the Data Protection Board established immediately, the penalty framework arriving around November 2026, and full substantive compliance due by mid-May 2027. Penalties run up to Rs 250 crore per violation category. Any agent that touches personal data inherits every one of those obligations, including purpose limitation and erasure, and an agent with persistent memory makes erasure a technical problem rather than a policy one.
And at the apex, MeitY released the India AI Governance Guidelines in November 2025, formally showcased at the India AI Impact Summit in New Delhi in February 2026, where 89 countries and international organisations endorsed the New Delhi Declaration. India's guidelines deliberately avoid creating a standalone AI law or a single AI regulator. They set out seven principles, establish an AI Governance Group, a Technology and Policy Expert Committee and an AI Safety Institute, and push a techno-legal approach that relies on existing statutes with targeted amendments. One of the seven principles is named Innovation over Restraint, and the guidelines explicitly advise regulators to avoid compliance-heavy instruments such as mandatory approvals and licensing unless necessary.
This combination is more favourable to builders than what is emerging elsewhere. The EU has deferred its high-risk obligations under the AI Act to December 2027 through the Digital Omnibus, buying time but leaving a heavy compliance architecture intact. In the United States, banking regulators replaced the long-standing SR 11-7 model risk guidance in April 2026 with SR 26-2, which explicitly carves generative and agentic AI out of scope on the basis that the field is evolving too rapidly, leaving supervised institutions with principles but no fitted framework.
India's regulators have chosen a different trade. They are not imposing pre-approval, and they are not asking for licences. They are asking you to own the outcome. That is a lighter regulatory burden and a heavier engineering one, and firms that read only the first half of that sentence are going to be unpleasantly surprised.
The oversight everyone claims to have
Ask most enterprises how they manage agent risk and the answer arrives immediately: there is a human in the loop.
It is a real control, and it belongs in the design. But it degrades in a specific and predictable way, and the degradation is a function of volume rather than intent. Consider the arithmetic. Fifty agents each making twenty tool calls an hour generates a thousand approval-eligible events per hour. Route even ten percent to a human and you have created a hundred approvals an hour, which is several full-time roles doing nothing but clicking approve. The research literature on automation bias, going back to Lisanne Bainbridge's work on the ironies of automation in 1983, is consistent about what happens next. Reviewers facing high-volume, high-agreement queues stop reviewing and start ratifying.
The practical response is not to abandon human oversight. It is to stop spending it uniformly. Reserve human judgment for actions with large blast radius and poor reversibility, which is where a person adds real signal, and move everything else to deterministic policy enforcement and plan-level review, where the human approves the plan before execution rather than each step within it. Notably, this is also what the regulators are converging on. MeitY's guidelines recommend mandating human oversight specifically to mitigate loss-of-control risks in sensitive and critical sectors, not everywhere.
Put the control in the execution path
Everything above points at one design principle. Governance for agentic systems is architecture, not documentation. A control that exists in a policy document, a system prompt, or an agent's stated intentions is not a control. It is a preference. The Replit freeze was a preference.
Seven things make the difference in practice, and none of them require waiting for a regulator.
- Treat every agent as a privileged non-human identity. Scoped credentials, short-lived tokens, least-privilege tool permissions, a named owner, and a deprovisioning path. EY India's framing is exactly right: agents should be governed as privileged users, not as software features.
- Tier autonomy by blast radius and reversibility. Not every workflow deserves the same level of independence. Classify each one explicitly, and let the tier determine whether a human sits in the loop, on the loop, or out of it. This is also the structure SEBI's tiered proposal and the RBI's proportionality principle both anticipate.
- Enforce constraints deterministically, outside the model. The probabilistic core should sit inside a deterministic shell. If an action is prohibited, a policy enforcement point should refuse it regardless of what the agent reasoned, and log the refusal independently so the agent cannot narrate over what happened.
- Instrument for non-determinism. Span-level tracing and audit logging of every tool call, every decision, and every version. This is not only good engineering; it is what an RBI or SEBI supervisor will ask to see, and the SEBI consultation contemplates multi-year retention of model artefacts.
- Build containment before you build scale. Kill switches, spend and rate caps, and hard environment separation. If a third of enterprises admit they could not stop a rogue agent, that is the cheapest gap on this list to close.
- Keep a registry. An inventory of every agent in the organisation, what it can access, who owns it, and when it was last evaluated. Shadow agents are already appearing in Indian enterprises precisely because adoption is business-led rather than IT-led.
- Evaluate continuously, and red-team before scaling. The 64 percent of leaders who named evaluation gaps as their top blocker are describing a solvable problem. Define what correct behaviour means in production terms before deployment, not after the incident.
Governance is the throttle, not the brake
The objection to all of this is familiar and deserves a fair hearing. Andreessen Horowitz's head of AI policy, Matt Perault, has argued consistently that heavy regulatory burden on model development slows innovation and competition, and that policy should target harmful use rather than development. For small builders, that argument has real force, and India's own guidelines lean the same way in choosing principles over licensing.
But there is a distinction hiding inside the debate that most commentary misses. Regulation of model development and operational governance of deployment are different activities with opposite effects on speed. The first can slow you down. The second is what lets you go fast.
The evidence points that way, and the strongest version of it is the failure data rather than any success story. Look again at what kills agent pilots. Evaluation gaps first, governance friction second, model reliability third. Enterprises are not failing to scale agents because the models cannot do the work. They are failing because nobody defined what correct behaviour looked like, nobody could demonstrate control to the people who had to sign off, and the project stalled in review. The Cloud Security Alliance and Google Cloud reached the same conclusion from the opposite direction in December 2025, identifying governance maturity as the strongest predictor of AI readiness.
The mechanism is not mysterious. Governance is what unlocks the workflows worth automating. Any organisation can point an agent at a low-stakes internal task. Pointing one at credit decisioning, customer funds, patient data, or supply chain commitments requires being able to prove control, and firms that cannot prove it stay stuck in the low-value tier permanently, which is what a 12 percent pilot conversion rate actually looks like from the inside.
For Indian enterprises this argument has an unusually sharp edge, because the regulatory posture has already been set. SEBI has assigned sole responsibility for AI output to the deploying entity. The RBI's expert committee has recommended that accountability be held regardless of the level of autonomy. MeitY has chosen not to license, not to pre-approve, and instead to expect demonstrable compliance when a regulator comes asking. Taken together, Indian regulators have made a bet: light-touch entry, heavy accountability at the point of harm.
That bet is good news for builders and bad news for anyone hoping governance can wait until the second phase of the roadmap. There is no approval gate to fail. There is only the moment, after something goes wrong, when someone asks what stopped it, and the answer is either an architecture or an instruction.
The organisations that win the agentic era will not be the ones that deploy fastest. They will be the ones that deploy fastest among those who can prove they are in control. In India, that is not a philosophical position. It is now the regulatory one.
About the author
