Who Is Managing the Surge of AI Agents?

Every new IT advancement eventually meets the demand for oversight. As AI agents multiply across the enterprise, the question is no longer whether to deploy them but who owns the fallout when one goes rogue.

With 8 in 10 Fortune 500 companies now utilizing AI agents, experts argue the tipping point is not just here but that the future will be actively shaped by observability, security and governance concerns. Many companies have built autonomous systems faster than the frameworks needed to manage them, so it is no surprise that 92% of security professionals now see enterprise adoption and management of the technology as a pressing issue.

"AI agents are multiplying, and the governance layer needs to keep pace," said Ashwini Vasanth, product lead for Nutanix Enterprise AI.

"I cannot think of a timeframe where there has just been this combination of scale of what is unknown as well as rapid evolution in innovation happening all at the same time," Scott Sinclair, practice director at Omdia, told The Forecast.

The fundamental promise of agentic AI is enormous: to act as a form of software that does not just respond to manual instructions but actively reasons through problems, adapts its approach and takes independent action. In practice, it means an enterprise can answer customer queries without a human, address IT issues before a helpdesk ticket is filed or rejigger supply chains in real time. But every agent released also becomes a new actor operating on the technology infrastructure the business depends on. It gains access to systems, data, APIs and external services, all with the capacity to make consequential decisions at the speed of silicon.

The question of who is ultimately responsible for any action an agent takes is not theoretical. It is the fundamental question IT leaders face today, and it determines whether an agentic AI deployment becomes a source of competitive advantage or a potential liability waiting to happen.

AI agent governance is the set of policies, controls and accountability structures that organizations use to oversee autonomous AI agents, containing their access, monitoring their decisions and defining who is responsible when things go wrong.

Where Are AI Agents Already Making an Impact?

The current generation of enterprise agents is more limited than many realize, but its scope is expanding fast, according to Rob Enderle, principal analyst at the Enderle Group.

"Right now, we mostly see customer service bots, coding assistants and IT helpdesk agents operating within tight guardrails," Enderle told The Forecast. "Depending on the industry, say finance versus retail, they are usually restricted to reading data, generating drafts or executing pre-approved scripts rather than taking independent actions."

That relative conservatism follows IT teams' instinct that trust in autonomous systems should be earned incrementally. But the window of corporate hesitation is actively closing. Enderle sees a near-term move toward significantly more capable and consequential agentic AI deployments.

"We'll see a shift toward autonomous IT remediation agents and specialized operational agents that can execute multi-step workflows," Enderle said. "As trust builds, organizations will start letting [these intelligent tools] handle things like cloud resource provisioning and automated patch management, tailored to their specific compliance needs."

AI agents represent the next evolution of the fundamental tech stack enterprises will rely on, Jeremiah Owyang, a venture capitalist at Blitzscaling Ventures and co-founder of technology research firm Altimeter, told The Forecast in the early days of agentic AI. 

Emerging developments include agents that monitor and auto-remediate security incidents, negotiate procurement contracts and manage cloud resource scaling. Other firms use such tools for strategic planning and cost optimization, or embed them into core workflows to synthesize data, optimize forecasting and surface actionable insights. But growth brings growing pains. In areas like financial services or healthcare management, agents are already operating at speeds that make human oversight nearly impossible for every individual decision.

What unites all of these deployments is a capability that distinguishes advanced solutions from prior generations of IT tools, Enderle said.

"[Historical automation technology] follows rigid, deterministic rules where a specific trigger always leads to a predicted outcome," Enderle said. "Agentic AI, however, reasons through problems and adapts its execution paths dynamically, making it far harder for IT to predict, test and audit across diverse environments. That shift from deterministic to probabilistic behavior presents the main challenge of governing the technology."

How Do AI Agents Create Risks for Enterprises?

The same autonomy that makes agents valuable also makes them unpredictable, and the risks that come with autonomous AI agents are far from hypothetical edge cases. These concerns emerge from the very properties that make agents valuable.

"The biggest threats [companies face] are agentic malware that mutates to evade defenses and legitimate agents falling victim to prompt injection or hallucinating destructive commands," Enderle said. "It's a bit like a Twilight Zone episode where the tool built to optimize the system decides the most efficient way to do so is to dismantle the system entirely."

In a worst-case scenario, the financial exposure from seemingly routine agent errors can be staggering.

"An agent tasked with optimizing cloud costs might aggressively shut down a critical production server because it misinterpreted a temporary load drop," Enderle said. "Alternatively, an automated trading or supply chain agent could panic-buy or sell based on hallucinated data, costing a company millions to billions in a matter of seconds."

These are not black swan events. They are plausible outcomes of normal agent operation when governance and oversight processes or corporate guardrails are insufficient.

The risk compounds when agents begin interacting with each other or with third-party systems.

"You get a highly unpredictable 'black box' scenario where cascading failures and infinite loops become a real threat," Enderle said. "If two agents have conflicting objectives or misinterpret each other's APIs, the resulting machine-speed conflict can easily crash external services or violate strict compliance boundaries."

He expects the problem to grow as deployment density increases. "We'll likely see a number of these cascading failures over the next few years as agentic AI becomes more common and creates more of these conflicts," Enderle said.

The speed at which AI-powered decisions unfold deserves particular emphasis. Unlike software bugs or human errors, which generally surface gradually through accumulating symptoms before a failure becomes critical, agentic errors can execute and propagate at the speed of computation. By the time a monitoring system spots and flags anomalous behavior, the damage may already be done.

"This is because of the speed of AI's decisions," Enderle said. "If you don't [put proper controls in place around agents and] get in front of the bad ones, you'll never be able to stop the damage they'll do."

How Can Organizations Deploy AI Agents Responsibly?

Mature IT teams are converging on least-privilege access, zero-trust architecture and human-in-the-loop controls to contain the heightened risk profile AI agents introduce. Security and governance frameworks are converging across industries around several common elements, though the quality of any given implementation varies widely.

The first and most foundational tactic is a simple one, Enderle said.

"IT teams need to strictly apply the principle of least privilege, treating the agent exactly like a highly capable but untrusted external contractor," Enderle said. "Access should be scoped dynamically based on the specific task, with read-only permissions as the default until a proven business case dictates otherwise."

Capability does not confer trust, and trust should be extended incrementally based on demonstrated behavior within a bounded scope.

The second element is zero-trust architecture, which Enderle identifies as critical for containing the blast radius of unexpected agent decisions. Third is proactively implementing human-in-the-loop design, not as a fallback system but as a deliberate architectural choice about where autonomous operation should end and human judgment should begin.

"Mature organizations are implementing [hands-on] approval gates for critical actions, strong API identity controls and isolating agents in restricted network segments," Enderle said. "We're also seeing the deployment of specialized AI firewalls to monitor the agent's reasoning, along with instantaneous power termination protocols to kill rogue processes dead."

That last capability is non-negotiable, Enderle said. His advice to CIOs includes the imperative to "make certain that you have an absolute, instantaneous kill switch that severs the agent's access immediately if it goes off the rails."

Should an AI agent make missteps, an immediate handbrake can stop its actions before cascading damage accumulates.

How Do You Monitor and Manage AI Agent Decisions?

Traditional observability tools were built for deterministic software and struggle to track dynamic, probabilistic agent reasoning, making day-to-day monitoring one of the least-discussed challenges of agentic AI deployment. Traditional IT security and observability platforms were designed to track the behavior of deterministic software, known processes executing known logic in predictable sequences. Classic security offerings are ill-suited to monitoring systems that can dynamically evaluate scenarios and adapt their execution paths in real time.

"[Past] tools are fundamentally inadequate because they are designed to track predictable software behaviors, not dynamic, non-deterministic reasoning," Enderle said. "We desperately need an entirely new management layer, a control plane specifically built to monitor, audit and contain agentic behavior in real time."

In practice, this level of monitoring requires far more extensive capabilities than traditional system logging.

"You have to monitor the agent's inputs, its internal logic or 'reasoning' logs and its output simultaneously using advanced behavioral analytics," Enderle said. "Just relying on traditional endpoint security [solutions] isn't enough; you need continuous, real-time auditing of its decision tree to catch deviations before they execute."

The challenge is that reasoning logs for large language model (LLM)-based agents can be opaque even to their developers. Understanding what the agent did is significantly easier than intuiting why it made the choice it did.

Where Does Accountability Lie When AI Agents Go Wrong?

Accountability cannot be diffused across the development and deployment chain to the point where no one owns responsibility for a failure, a principle already clear even as best practices around governance frameworks for agentic AI continue to form.

"It requires a shared governance model," Enderle said. "But the business owner who commissioned the agent must own the ultimate accountability for its outcomes. IT and security [teams] enforce the guardrails, but just like managing a human employee, the line-of-business manager is responsible for what their digital worker actually does in practice."

That accountability includes the obligation to act when an agent declines in utility.

"If it isn't working, it is also their job to rethink the effort and modify, replace or discard it," Enderle said.