Introduction
Coding agents saw rapid adoption within Nutanix. Within a year of adopting a commercial AI coding assistant, almost all of the company’s roughly 4,000 engineers were using it. That included software developers, QA teams, and site reliability engineers. They used it to write code, review pull requests, chase down bugs, and work through unfamiliar parts of the codebase. Productivity rose, and so did expectations of what an engineering team could get done in a sprint.
Then the pricing changed. The vendor moved from metering via API to charging per token. At the time, Nutanix engineers were consuming tokens at a massive scale, leading to rapidly escalating cost projections for the engineering workload alone. Usage was still climbing. Engineering leadership saw a cost curve with no ceiling and decided to look for another way.
The answer was an internal AI token factory. Nutanix spent $20 million on GPU servers and built a private inference platform on its own software stack. The platform serves open-weight models to developers through a single governed gateway. The company expects the investment to pay for itself within one to three years. It also gave Nutanix a working blueprint for something many enterprises are now trying to do, which is to run AI as a shared internal service with costs that can be forecast, access that can be governed, and usage that can be measured.
Why Agentic Workflows Break Per-Token Budgets
A chatbot exchange might use a few thousand tokens. An agent working on a single coding task behaves very differently. It might inspect a repository, pull documentation, generate code, run tests, read the errors, revise its work, and ask follow-up questions before it finishes. Each step can trigger several model requests. When the agent spins up sub-agents to work in parallel, or passes the same context back and forth because nothing is cached or compacted, token counts can grow rapidly.
The result is inference traffic that grows much faster than the number of people using it. Token consumption can grow faster than user counts when engineers rely heavily on agents, parallel sub-agents, or repeated context. Under per-token pricing, that growth goes straight into the bill, and the bill is hard to predict from month to month.
Published benchmarks and prices make the case for ownership. Eight-GPU NVIDIA B300 servers sell for about $400,000 to $650,000 each, and every $1 million of hardware costs $250,000 a year over a four-year span1. On the previous-generation B200, vLLM benchmarks show 15,000 to 35,000 tokens per second per GPU with an open model such as gpt-oss-120b2. At that speed and only a third of capacity, each $1 million of hardware could process an estimated 170 to 610 billion tokens a month3. For example, using Cursor's Composer 2.5 API rates, about $0.72 per million tokens for that workload, the same volume would cost an estimated $1.4 to $5.3 million a year, roughly 6 to 21 times the yearly hardware cost before power, cooling, and operations4. Because the hardware cost is fixed, running the servers more of the time lowers the cost per token further, while API costs rise with every token.
An 80/20 Split Between Open-Weight and Frontier Models
Owning inference does not mean giving up frontier models. Some tasks still call for the deepest reasoning available, and those models are only offered as cloud services. Nutanix set an internal target of generating approximately 80% of its tokens on self-hosted open-weight models, with the other 20% to frontier models through cloud endpoints when a task needs them.
That split shaped the design. The platform was configured to make the selected on-premises models the default in this deployment, without cutting developers off from cloud models. It also had to keep cloud usage from quietly growing back into the problem the project set out to fix. Rate limits are applied at the gateway to restrict the number of requests per minute to cloud models, preventing overuse and keeping costs in check.
What Sits Underneath the Token Factory
The token factory runs on the Nutanix Agentic AI solution, which combines infrastructure operations with an AI platform-as-a-service for building, running, and protecting agentic applications. Five parts of the portfolio do most of the work:
- Nutanix AHV, the company’s hypervisor, virtualizes the GPU nodes. It delivers near bare-metal performance while keeping workload isolation and the life cycle operations that come with the Nutanix virtualization stack.
- Nutanix Kubernetes Platform (NKP) is used to run the clusters hosting the AI services.
- Nutanix Enterprise AI provides two key capabilities, Nutanix Agent Gateway for governed model access and Nutanix Private Inference for model serving.
- Nutanix Unified Storage is configured to hold model artifacts and selected metrics and logs collected by the platform.
- Nutanix Database Service is used to run a managed PostgreSQL instance in each environment to store application state, model definitions, endpoint configurations, and identity data.
The internal deployment described here used six Nutanix clusters. A management cluster hosts Prism Central, the NKP management plane, Agent Gateway, and the observability stack. A dedicated storage cluster runs Unified Storage. The GPU capacity sits in two types of AI compute clusters. One type is built on Cisco MGX nodes with NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. The other uses Supermicro HGX nodes with NVIDIA B300 GPUs. Each type is deployed in pairs for high availability. The MGX clusters carried the pilot phase. The HGX clusters are planned for the production rollout, and once that is done, the MGX nodes will be moved to smaller models while the largest models run on HGX.
Leveraging workspaces in NKP helps keep this manageable. Each workspace groups clusters so that configurations, policies, and access controls can be applied the same way across them. That matters because the MGX and HGX clusters need different GPU and network operator settings, and because the same Helm chart is used with the gateway turned on in one place and turned off in another. Separating them into workspaces lets the team apply hardware-specific settings without managing each cluster by hand.
Agent Gateway as the Control Point
Most of what makes the token factory work as a shared service happens in Nutanix Agent Gateway. Every model request from a developer passes through it, whether the model is running on-premises or in a public cloud.
For developers, Nutanix built a coding harness, a managed runtime, and launcher for the OpenCode agent that connects to approved model endpoints through the gateway. The configured developer workflow provided named endpoints and an API key scoped to the Nutanix Enterprise AI environment. Credentials for the underlying providers stay inside the gateway and never reach the developer’s machine or the agent.
Administrators work with unified endpoints. The configured endpoints route to selected models from one or more providers. An endpoint can run in load balance mode or fallback mode. In the pilot, three endpoints were backed by open-weight models running on the two MGX clusters, including NVIDIA Nemotron 3 Super and Poolside Laguna. Each endpoint splits traffic evenly between the two clusters, allowing maintenance of one cluster without requiring developers to change the endpoint they used. Larger models such as Laguna run across multiple GPUs using tensor parallelism and KV cache-aware routing to keep throughput up.
A fourth endpoint routes to a frontier model on AWS Bedrock for the hardest reasoning work. The configured quota and routing policies were intended to support the 80/20 internal target. Each client is limited to 25 million tokens a day on that endpoint. In the configured workflow, clients exceeding the daily limit received an HTTP 429 response and were directed to an on-premises model until the quota reset. Fallback mode offers another option. An endpoint can be set to try a self-hosted model first and fall back to a cloud provider only if the local model returns an error.
The abstraction also protects developers from churn. Open-weight models are improving fast, and Nutanix already has several new ones lined up for testing. When an administrator swaps the model behind an endpoint, the endpoint name and the developer’s connection stay the same. The harness keeps working, and engineers get a better model without changing anything on their end.
Visibility Across a Hybrid Environment
Cost analysis depends on visibility into token usage and related inference metrics. Agent Gateway records detailed metrics for every endpoint, whether it is backed by a cloud provider, a self-hosted model, or both. Those metrics follow OpenTelemetry standards, so they can be exported to whatever monitoring tools a team already uses.
Nutanix sends the data to VictoriaMetrics and builds dashboards in Grafana. The team can see token usage by endpoint and client along with inference engine metrics such as latency and time to first token. Unified Storage keeps the aggregated metrics and logs for long-term retention. That history is what lets the company check its ROI model against real consumption, plan GPU capacity ahead of demand, and spot workloads that use more tokens than they should.
The Platform Design and Operations Challenge
Buying GPUs was the easy decision. Building a reliable service on top of them took coordinated work across the company’s IT, AHV, NKP, Nutanix Enterprise AI, and developer relations teams. Standing up a production inference environment means getting dozens of interdependent settings right across hardware, virtualization, Kubernetes, model serving, and networking. It also means keeping all of those components upgraded and compatible over time.
This is where the Nutanix approach pays off for teams trying something similar. Organizations that assemble an AI factory from separate components end up owning every dependency, every upgrade, and every integration point. The Nutanix stack aims to cover the full life cycle of this deployment. Foundation handles initial deployment. Nutanix Life Cycle Manager keeps the software components current. NKP and its application catalog manage the dependencies of the AI services running on top. It is the same operating model Nutanix brought to hyperconverged infrastructure, applied to AI workloads.
The Nutanix experience is not unusual. Across the software and technology industry, major enterprises are facing rapid increases in AI token costs, leading many to ration AI access and redirect budgets toward more cost-effective solutions. Agentic tools prove their value, adoption takes off, and per-token pricing turns that success into a budget problem. Some organizations may consider customer-controlled or on-premises inference where data-location, access-control, contractual, or jurisdictional requirements are relevant.
Nutanix built the token factory to solve its own problem. The project also produced a practical model for other enterprises. Run most inference on open-weight models in infrastructure the company controls. Keep frontier models available for the work that needs them. Put a governed gateway in front of everything so policy, routing, and cost tracking happen in one place.
Organizations that follow this model can give their teams as much AI as they can put to good use, with a potentially more governed approach to cost planning and data-location and access controls, depending on the selected deployment and operating model. As agents take on more of the work, this approach may help organizations develop a more governed operating model for scaling AI workloads.
For a deeper look at the cluster design, the NKP and Nutanix Enterprise AI configuration, and the observability setup, read the Nutanix Agentic AI Token Factory solution document.
1 Published prices for a new eight-GPU NVIDIA HGX B300 server range from $400,000 to $450,000 (AMCompute, October 2026 Rack Report, https://www.amcompute.com/gpu-server-values/b300) to $550,000 to $650,000 (Thunder Compute, October 9, 2026, https://www.thundercompute.com/blog/nvidia-b300-pricing). Actual prices depend on configuration, volume, and vendor. Across that range, $1 million buys about 12 to 20 GPUs. Lower prices widen the cost advantage of ownership; higher prices narrow it.
2 SemiAnalysis InferenceX, gpt-oss-120b on NVIDIA B200, vLLM, FP4 precision, single-turn workload of 8,000 input and 1,000 output tokens per request; runs dated February 5 to May 31, 2026. The range spans 15,470 tokens per second per GPU at about 200 tokens per second per user to 34,672 at about 75 tokens per second per user. Benchmarks were run on the B200, not the B300. No B300 results for this model had been published as of November 5, 2026.
3 Calculated as 12.3 to 20 GPUs per $1 million × 15,470 to 34,672 tokens per second × 2,628,000 seconds per month (30.4 days) ÷ 3, or about 167 to 607 billion tokens. The low end pairs the highest server price with the slowest benchmark speed; the high end pairs the lowest price with the fastest speed. The one-third utilization rate is an assumption, not a measured figure. The benchmark does not state whether throughput counts input and output tokens; its reported concurrency and per-user speeds indicate that it does. Results on other models and workloads will differ.
4 Cursor's published Composer 2.5 rates as of November 5, 2026: $0.50 per million input tokens and $2.50 per million output tokens. Cursor applies these rates to usage beyond the weekly pool included in paid plans, and negotiated enterprise rates may differ. The average assumes the benchmark's ratio of eight input tokens per output token: (8 × $0.50 + $2.50) ÷ 9 ≈ $0.72. No discount for cached input ($0.20 per million) is applied; workloads with substantial cached input would cost less. Annual cost is 167 to 607 billion tokens a month × 12 × $0.72 per million. The $250,000 yearly hardware cost is $1 million spread over four years and excludes power, cooling, datacenter space, networking, and staff. The comparison assumes comparable output quality between Composer 2.5 and gpt-oss-120b, which differ in design and purpose.
©2026 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and all Nutanix product and service names mentioned are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. Kubernetes is a registered trademark of The Linux Foundation in the United States and other countries. All other brand names mentioned are for identification purposes only and may be the trademarks of their respective holder(s). This content reflects ongoing internal development and experimentation. Results, benefits, savings, or other outcomes described depend on a variety of factors including use case, individual requirements, and operating environments, and this publication should not be construed as a promise or obligation to deliver specific outcomes.