Building the Enterprise Foundation for Agentic AI

Enterprise AI has crossed a threshold. A year ago, most boardroom conversations centered on proving return on investment and figuring out where generative AI might fit. Today the question has changed. Organizations are no longer asking whether AI belongs in the business. They’re asking how to build the infrastructure, governance, and operating model that let autonomous agents act, orchestrate, and make decisions at scale.

This shift from chat-oriented AI to action-oriented AI will be the defining story of the next several years. Coworkers, autonomous agents, and coding assistants are already driving enormous demand, and coding in particular is pulling ahead as a proving ground. Development teams are moving away from traditional software lifecycles toward agent loops, where models iterate continuously on a problem until they land on an optimal result.

The pattern is powerful, and it is also expensive. An agent given a week to optimize a piece of code, or to work through a materials science problem, can burn through an extraordinary volume of tokens before it converges. That tradeoff – of speed and autonomy against cost – sits at the center of nearly every conversation executives are having right now.

The Cost Reckoning

The early phase of agentic AI looked a lot like the Wild West. Teams threw powerful models at problems just to see what would happen, and token cost visibility took a back seat to experimentation. CFOs have begun to take notice, and that scrutiny is a good thing. It’s pushing organizations toward greater discipline, driving them to optimize token usage and AI infrastructure rather than defaulting to the largest available model for every task..

Not every use case needs a frontier-scale, state-of-the-art model. A customer service agent following a defined workflow, for instance, may perform just as well on a smaller, fine-tuned model as it would on a far larger one. The emerging discipline is matching model size and capability to the specific job, then layering in the growing ecosystem of open source and open weight models to keep more of that decision, and more of the cost, under enterprise control.

This is quickly becoming a deterministic exercise. Leaders identify the class of problem, select the appropriately sized model, and manage the outcome rather than letting a single sprawling model run indefinitely against an open-ended budget.

Governance Cannot Be an Afterthought

Security and governance rank among the top concerns for any organization deploying agentic AI, and for good reason. Left unmanaged, sensitive data and intellectual property can leak into frontier models in ways that are difficult to trace or reverse. Regulated industries face even higher stakes, where a governance gap can translate directly into liability.

Solving this requires more than policy. Organizations need a control layer that sits between agents and the underlying infrastructure, one that helps access controls, applies guardrails around what individual agents can touch, and is designed to help protect personally identifiable information by design.

This is genuinely hard infrastructure to build, and it represents one of the clearest opportunities for platform providers to deliver lasting value. A gateway is designed to give organizations that govern agents, tools, and models greater control over how agentic applications get built and deployed, rather than leaving governance to trail behind adoption.

Managing Complexity Without Slowing Innovation

The agentic AI stack today can resemble a Rube Goldberg machine, with many components, from observability to context management to agent orchestration, that must work together seamlessly. Many modern organizations find themselves somewhere between throwing workloads at frontier labs and assembling bespoke pieces in-house, and neither extreme is sustainable long-term.

A hybrid, multicloud architecture is emerging as the practical answer. Platforms with deep, established roots in enterprise data are well positioned to manage the access control layers, telemetry, and guardrails that agentic AI demands. They also give organizations the flexibility to burst into the cloud when they need scale, while keeping sensitive workloads on-premises when they need control.

Open weight models play a meaningful role here too. They have kept remarkably close pace with closed alternatives, providing a credible path to air-gapped, privately controlled inference for sensitive data, while still tapping frontier models for the reasoning-heavy tasks that benefit from them.

A Mental Model for What Comes Next

One useful way to think about the coming architecture is to separate the agent from the inference layer. Agents, running in harnesses and sandboxes, increasingly built on micro VM architectures with strong security controls, are best co-located with the business, the workers, and the data they support. Inference is where additional intelligence gets applied, and it does not have to live in a single place.

Organizations should be able to draw on the optimal available option, whether that’s a public API, on-premises model, or hybrid cloud combination, without waiting on GPU availability to get started, while still retaining the option to procure dedicated capacity as needs mature.

This bimodal approach reflects where the technology is truly headed. Cost pressure, data sovereignty, and the need for control are not going away, and no single environment—whether public cloud, private data center, or the edge—will satisfy all three simultaneously.

Coding is already showing the way. Many organizations reserve frontier models for the more complex parts of the workflow while keeping code generation on models they can control, in an environment that keeps the code inside their own perimeter.

None of this changes overnight. Organizations are still in a discovery phase, working out which architectures, models, and governance structures hold up at scale. But the outline of a durable, hybrid, well-governed approach to agentic AI is becoming clear, and CIOs who start building toward it now may be far better positioned for the wave of adoption still ahead.

For more information, visit https://www.nutanix.com/enterprise-agentic-ai.