Run Your AI in the Cloud. Or Don't. That's the Point.

Introduction

Almost every enterprise AI project starts in the cloud. Somebody gets an API key, points a retrieval pipeline at four years of support tickets, and has something demoable inside a couple of days. No capital request. No procurement cycle. No GPU. A purchase order can't tell you whether the use case is real. Renting a model for a week can. So the cloud wins round one by default, and it deserves to. AI in the cloud is where you go to find out cheaply and fast.

Round two arrives about six months later, when the thing works, volume is climbing, and somebody senior asks where this is going to live at scale. Or you look up and you're staring at a six-figure AI bill. The answer might be the cloud. It might be the estate you already own and are already depreciating. What it shouldn't be is a question you answered by accident in week two when you picked an SDK.

Nutanix gets filed in a lot of people's heads as the hyperconverged infrastructure company: the thing in the rack, an on-prem answer to an on-prem question. Fair, given most of a decade of doing exactly that. The preconception is what costs people the better conversation, because they assume Nutanix isn't in round one at all.

Nutanix can help with AI workloads with one platform and manage them in the cloud or on-premises.The application and agent tier is CPU work, long-running and stateful. The inference tier is GPU work, expensive and spiky. The Nutanix Kubernetes Platform (NKP) solution manages the fleet those workloads run on, and the Nutanix Enterprise AI (NAI) solution offers two products: the Nutanix Agent Gateway solution and the Nutanix Private Inference service on top, and both offer a consistent experience whether that fleet is sitting in AWS/Azure/Google or in a building you own.

Nobody buys a GPU to find out

Consider what round one looks like if you insist on doing it properly, on-prem, from a standing start.

You need GPUs, which means a purchase with a real lead time attached. You need somewhere to put them, which means rack space, power and cooling that somebody has to sign off. You need the driver and CUDA stack, an inference runtime, a model store, and a Kubernetes layer underneath the whole thing. Optimistically you're a quarter out and well into six figures before anybody in the support organization has typed a question into the prototype.

You'd be spending all of that to answer a question that comes back "no" a reasonable fraction of the time. Most AI pilots don't graduate. That's what a pilot is for, and it's an entirely sane reason to refuse to fund infrastructure in front of the evidence. The cloud removes that whole problem. Rent the GPU by the hour, run the experiment, kill it on a Friday if the answer is no. Nobody has to defend a capital line item for a test.

The trap is what happens when the answer is yes, and it's a portability trap rather than a technical one. If round one got built on a hyperscaler's agent framework, its knowledge bases and its guardrail system, then the model call is portable and nothing around it is. Bedrock is AWS. Vertex is GCP. Azure OpenAI is Azure. Each brings its own SDK, its own agent abstractions, its own guardrail configuration. Moving the workload means rebuilding the operational surface you spent six months getting right, which is expensive enough that most teams simply don't, and the location decision quietly gets made for them by sunk effort.

Your proof of concept may not be that costly. The quarter-and-six-figures estimate assumes you're building from nothing, which is true for a company with no estate to speak of. It looks different if you already run Nutanix. The proof-of-concept ask on an existing cluster is one GPU node, not a GPU fleet: NKP creates a GPU node pool at a single replica with a single card attached, the Nutanix Private inference will map the hardware to the model and context length requested to bring up an operational inference endpoint. No guess work. In addition, there is a validated model catalog that segments openweight models by use case and size ranging from those that can run on CPUs to those that need multiple GPUs.

NKP's documented cluster minimum is four worker nodes on Pro and Ultimate, and NAI wants three workers of its own for its control plane before the GPU pool goes on top, so the single-node Kubernetes cluster is not a thing to plan around. The single node GPU has the lead time and the price tag is a quantity of one. The rest is CPU capacity you probably already have racked.

The Nutanix answer to the portability trap isn’t sexy and, I think, the whole point. NAI has tons of deployment options:

• NAI on EKS/AKS/GKE
• NAI with NKP on EC2 
• NAI with NKP on Nutanix Cloud Clusters (NC2)

NKP with the full-stack packaging ships alongside NCI, NDB and Nutanix Unified Storage. Same Hugging Face import workflow, same security scan on the way in, same OpenAI-compatible endpoints coming out. NKP fleet-manages the clusters underneath in either place and the applications. So the thing you validated in round one is the thing you deploy at scale in round two, in whichever building the numbers point to, and the migration is a deployment target rather than a rewrite.

The same platform in two places

The Nutanix Cloud Clusters (NC2) platform provides the infrastructure layer: Nutanix AHV hypervisor clusters on dedicated bare-metal hosts with the Nutanix storage fabric underneath. The stateful CPU side of the RAG application lives there, on an NKP workload cluster, because the document ingestion pipeline is exactly the kind of thing that wants queues and persistent volumes on storage that can snapshot and replicate them. The GPU side runs NAI on an EKS or NKP on EC2 cluster of p5 nodes. The app and API tier sits on a non-GPU EKS cluster next to the retrieval store. Every one of those clusters, the NC2-hosted ones and the EKS ones alike, reports to a single NKP management cluster.

That's the real reason the inference layer here sits on EKS/EC2 p5 nodes instead of on NKP-on-NC2. NC2 on AWS doesn't have a GPU option suited to AI inference today. If that changes this can be revisited, because the supported instance types change constantly.

So the umbrella claim is that Nutanix on cloud covers both compute shapes, CPU workloads on NC2 and GPU inference on an attached EKS cluster, under one management plane. Nutanix is not running the H100s in this design. NKP is fleet-managing the cluster that does, which is an NKP Ultimate capability, and NAI is running the inference service on top of it. On-prem, where you choose the cards, both shapes collapse onto infrastructure you already own, with NKP Ultimate governing clusters in both places from one control plane. The management model doesn't change when the location does. NKP Full Stack solution on prem will give you the ability to deploy object storage and database as a service to complete your full AI solution.

Most of your company doesn't need a frontier model

Enterprise AI buying tends to assume everyone needs the best model on the market. Watch what people actually ask an internal assistant. Where does the expense policy live? Summarize this ticket thread. Draft a reply to this customer. What changed in this config file last week? That is not frontier work. A well-served open model in the 70B class handles all of it, and provides performance comparable to frontier models for many common queries.

Look at the great performance coming out of GLM 5.3 in the picture below. These open models are really catching up and exceeding frontier models in some cases.

The queries that genuinely need a frontier model are real and they are a minority. The gnarly contract review, the architecture question carrying fifty pages of context, the case where being wrong is expensive enough to justify the rate.

NAI through the Nutanix Agent Gateway gives you a unified endpoint is a single API address that fronts multiple inference services, including ones running on different clusters. Behind it you can place your local endpoints, and you can place external providers: Anthropic, AWS Bedrock, Azure OpenAI, Cohere, GCP Vertex AI, Google Gemini, Mistral and OpenAI are all supported provider types, with the credentials managed centrally. There is also a Remote NAI provider, which fronts another NAI instance. With a Remote NAI provider you can put an on-prem NAI and a cloud NAI behind one endpoint, with the application talking to the endpoint and not caring which side answered.

Configuration offers two strategies: Load Balance, where you assign a load value per model, and Fallback Order, where you set the sequence NAI tries them in. Those are traffic distribution and failover. Neither one is semantic routing. NAI is not reading the question and deciding it looks hard enough to deserve Claude. Choosing which queries earn the expensive model stays your application's job: a routing rule you wrote, sitting in front of two endpoints, rather than one endpoint being clever on your behalf. Today there are lots of agent harnesses that are providing a top tier model for planning another cheaper model for building and even introducing a third for verifying the results so you can continue with a loop to finish your projects with hands off for the most part. 

Route the bulk of a company's questions to capacity you have already bought at a fixed price, escalate the minority that needs frontier capability, and your exposure to the pricing meter shrinks to that minority. The caching trap can only bite on what you send out the door.

Nutanix Agent Gateway, part of the NAI portfolio, can provide granular token usage observability as well as controls to set rate limits and budgets per user and agent. Unified endpoints support both global token limits as well as granular limits per API key to accommodate per employee or agent token budgets per month.

Why the hybrid cloud and not a cloud pitch

If I pitched you Nutanix purely as a way to run AI in the cloud, you'd have an obvious and fair response ready: AWS already does that, it's their cloud, they have a large team pointed at exactly this, so why would I add a vendor to it. That's a good objection, and it doesn't fully lose.

The hybrid framing changes what's being asked. Can one platform be where the workload lives on both sides of the move from experiment to scale. A hyperscaler wins the first half of that comfortably. None of them has a credible answer to the second half, and not for lack of engineering talent. They don't operate the other building.

That still requires the cloud half to stand up as a place you'd actually run. If it can't, the hybrid story is just a more elegant way of losing round one. So here is the cloud half, including the parts that do not flatter the pitch.

A Token meter can be guess

Start with what you're actually comparing against, because managed inference is priced in a way that makes comparison harder than it looks.

Per-token pricing feels like the honest model. You pay for what you use, no idle capacity, no shelfware. For a pilot it genuinely is the right call, which is most of why round one happens there.

The problem shows up at the boundary between the pilot and the platform. A per-token price is a rate, and a rate multiplied by an unknown is still an unknown. Nobody in the room can tell you how many tokens the support team will burn once the assistant is embedded in the ticket workflow, or what happens to average context length when someone decides retrieval should pull eight documents instead of three. Those are product decisions, made by people who have never seen the invoice.  A meter can hand you a number nobody in the room predicted. It's a reason why a lot of companies have reverted from full frontier models. I call this the million dollar surprise.  The Nutanix Agent Gateway can help here, providing visibility into the input and output token usage per user and agent.

The caching trap

Prompt caching is where this gets specific enough to be uncomfortable.

Bedrock prices Claude Sonnet 5 input three different ways. Standard input runs $2.00 per million tokens. Writing a prompt prefix into the cache the first time runs $2.50 per million, a 25% premium. Reading that same prefix back later runs $0.20 per million, a 90% discount. Output stays flat at $10.00 per million regardless. The same shape shows up across the market: GPT-5.6 Sol at $5.50 standard, $6.875 to write, $0.55 to read. Grok 4.3 at $1.25 standard against $0.20 cached.

That structure is a bet, and the bet is prefix reuse. If your workload has a long, stable system prompt that gets called over and over, you pay the premium once and collect the discount forever after. The economics are terrific. This is the scenario the published per-token comparisons implicitly assume.

Now consider RAG, which is what most enterprise AI in the cloud actually is. Every request carries a different set of retrieved documents. The prefix changes on every call. You pay the 25% write premium each time and you never once collect the 90% discount, because there's nothing sitting in the cache that the next request wants.

Run that to its conclusion. At zero prefix reuse, your effective input rate is $2.50 per million against a $2.00 no-caching rate. You switched on a cost-saving feature and made every input token in your application 25% more expensive. There's no alert, no anomaly, no line item labeled "you are losing this bet." Cache writes are a legitimate charge for a legitimate operation that completed successfully. The bill just runs higher than the arithmetic you did in the planning meeting, and the reason is a locality property of your retrieval layer that nobody thought of as a pricing input.

Two point to be aware of:

• Bedrock doesn't offer prompt caching on Meta's Llama models at all. Its pricing page lists on-demand and batch only, no cache columns. So if the open model you'd consider self-hosting is the one you're renting today, this particular trap can't bite you. You're already on the flat rate in both directions.
• The self-hosted side has its own version of this. Nutanix Private Interface has  automatic prefix caching that does the same trick with KV state, and documentation is clear that it brings no gain when new queries don't share a prefix with existing ones. Benchmarks are harsher than the docs: SqueezeBits measured roughly a 36.7% throughput reduction and about 25% higher time-per-output-token at high concurrency on random data with no shared prefixes, running Llama 8B on an A100. Hash table lookups and cache bookkeeping cost something, and when the hit rate is zero you pay the bookkeeping for nothing.

On fixed hardware, a bad caching decision doesn't add a charge. It eats throughput. Your bill doesn't move and you find out at capacity planning rather than at invoice time. That's still a real cost. It just isn't a surprise, because the ceiling was set the day the nodes were racked.

Numbers you can count before you deploy

Here's what capacity-based licensing looks like when you actually read the paperwork.

NKP is licensed per physical CPU core on bare metal and per virtual CPU core on VMs and public cloud. Control plane nodes don't need licenses. Fleet-managing an EKS cluster makes it an Attached Cluster, which requires NKP Ultimate, and the Cloud Kubernetes SKU that covers those EKS worker vCPUs sits on top of base Ultimate rather than replacing it. Two SKUs, both additive, both counted in vCPUs.

NAI has two licenses. Nutanix Private Inference can be licensed either with a vCPU license or GPU GB. Nutanix Agent Gateway is licensed based on the number of access keys which only changes when new users are introduced that need to be granularly observed or cost controlled.

That failure mode is the tell. When a capacity license runs out, things stop. That's disruptive, occasionally embarrassing, and completely visible. When a meter runs, things keep working beautifully and the number keeps climbing, and nobody notices until somebody in Finance asks a question in the wrong month.

Put real hardware behind it. The reference design I've been working through runs three p5.48xlarge nodes in an N-1 arrangement, so any two of the three carry full load if one dies. Each node is 192 vCPUs, which is 576 vCPUs across the EKS GPU cluster, plus 24 more for the NKP management cluster. Six hundred vCPUs. Each node carries eight H100s at 80 GB of HBM3 apiece, so 24 GPUs and 1,920 GB of GPU memory sitting behind the Nutanix Private Inference endpoints.

Hard pricing numbers come from the instance spec sheet. Neither one moves when the assistant gets popular, when somebody raises the retrieval count from three documents to eight, or when a product manager discovers agentic workflows. You can write them on a whiteboard on day one and they'll still be true in month eleven. 

A 70B open model on one of those 8xH100 nodes runs four replicas at two-way tensor parallelism, around 9,200 output tokens per second each and something near 36,800 aggregate. At the 300-output-token requests this design assumes, that's in the region of 440,000 requests an hour out of a single node, if the node is busy. If it isn't, you bought idle H100s. Fixed capacity is only cheap per request when the requests actually arrive. Run one replica where four would fit and every request you do serve gets more expensive, until a hyperscaler would have been the kinder bill. Hyperscalers are very good at not charging you for idle time. At low or bursty volume, that wins.

Push the other way and it inverts. Once demand is heavy enough and steady enough to keep the cards fed, the per-request cost of hardware you've already paid for keeps falling, while the meter charges the same rate on the millionth call as it did on the first. Somewhere between those two states the lines cross.

I am not going to give you the request count where that happens. Any number I printed would be this model, at this token ratio, on this GPU class, at this utilization, and every one of those is a variable in your environment. Change the model and it moves. Change the ratio of input tokens to output tokens and it moves again. The slope is the thing worth carrying: at pilot volume, rent. At volume that keeps expensive hardware genuinely working, running the model yourself starts to win, and it wins by more the busier you get.

Take the two numbers off the spec sheet, then get a Nutanix salesperson to put real figures against them. Your token volumes, your model, your GPU class, your utilization, and a quote. That's the only version of this arithmetic worth showing a CFO, and it's an afternoon of work rather than a project.

Model freedom

Model freedom is the one people underrate. NAI's pre-validated catalog covers roughly 60 models across 14 providers, each entry carrying the minimum GPU count for a given GPU model so you're not guessing at fit. Outside that, a wide variety of Hugging Face models you can legally deploy go through the import path and the security scan, plus NVIDIA NIM microservices through NGC. Fine-tuned variants serve without Provisioned Throughput lock-in. Bedrock's catalog is bigger by count and leans heavily on commercial API-only models whose weights nobody is going to hand you, though as the unified endpoint covers, you can reach those through NAI regardless. Bedrock covers the supporting categories too, just more thinly: one reranker against NAI's several, Titan embeddings against a wider open field, multimodal models that read images rather than the dedicated vision models NAI carries, and content safety as the configurable platform feature I mentioned earlier rather than guard models you deploy and swap like any other endpoint.

Then there's a compliance detail that deserves more attention than it gets. Bedrock supports VPC endpoints through PrivateLink, traffic stays off the public internet, and prompts aren't stored or used for training by default. Good. But inference profiles route. A geographic profile can send a prompt to another region inside the same geography, and a global profile routes worldwide. If your contract says data stays in Frankfurt, then your profile selection is the contract, and it's a configuration setting one deploy away from being wrong. NAI in your own VPC with no internet gateway has no path out at all. You hold the KMS keys, the network ACLs and the audit retention. I suspect this version will ramp with our Government Cloud clusters for customers looking for solutions to help meet their sovereignty needs. 

Running the hardware provides opportunities for predictable forecasting, which is the quieter half of what capacity licensing buys. Somewhere between the pilot and the third business unit onboarding, finance wants what this costs, full stop, for the fiscal year, in a slide with your name on it. A meter can't answer that. A salesperson with your volumes can.

Where it makes sense

I'm not going to tell you to run your AI on-prem. Start in the cloud. Nearly everybody should, for the reason at the top of this piece: it's the only way to learn whether the use case is real without paying for the answer first. I'm also not going to tell you the cloud is a trap. Plenty of workloads never get busy enough to justify owning the hardware they'd run on, and for those the meter is simply the cheaper answer.

Whatever you validate in round one should be the thing you scale in round two, in whichever building the economics point to, without a rewrite in between. NKP fleet-manages the clusters either way. NAI serves the endpoints either way. The workload doesn't much care which side of the fence it wakes up on, and neither does the team operating it, because the console and the import workflow and the endpoint contract are the same ones they learned six months ago. People file Nutanix under bare metal and move on. They miss that the rack was never the edge of the product.

I wrote a while back that your agent doesn't need a GPU, it needs a babysitter, and that piece ended on a line I've kept chewing on: the GPU is somebody else's API. It still is, and for round one it absolutely should be. Just make sure whose it is stays a decision you get to make in month six, rather than one that quietly got made for you in week two.

©2026 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and all Nutanix product and service names mentioned are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. Kubernetes is a registered trademark of The Linux Foundation in the United States and other countries. All other brand names mentioned are for identification purposes only and may be the trademarks of their respective holder(s).