How Nutanix Built a Private AI Factory to Control Token Costs

To eliminate unpredictable cloud token meters and service these AI coding workloads, Nutanix deployed a full-stack, on-premises AI Factory designed to act as an internal token generation engine. Nutanix’s strategy combines self-hosted, open-weight models with external frontier models, giving the company flexibility in how it manages AI costs, performance, and data privacy. The approach centers on four key elements:

• Hybrid 80/20 Routing Strategy: The team allocates 80% of routine, high-volume AI coding tasks to open-weight models hosted on its own private AI cluster to enable predictable costs with no per-token bill. They allocated the remaining 20% of complex, high-reasoning tasks to external frontier models in the public cloud.
• AI-optimized Infrastructure: Nutanix uses its full-stack Agentic AI solution to run self-hosted AI workloads on its infrastructure.
  
  
  • Nutanix Enterprise AI (NAI): Enterprise-grade platform for managing and scaling AI workloads across hybrid environments. It hosts and serves on-prem, open-weight models with enterprise-grade inference optimizations through Private Inference, and controls AI traffic through Agent Gateway, giving IT teams a single front door to enforce token rate limits and quotas, tool access, role-based access controls, and audit logging capabilities.
  • Nutanix Kubernetes Platform (NKP): Abstracts away the complexity of Kubernetes to give Nutanix AI teams a reliable, production-ready environment. Deep integration with the underlying infrastructure automates node provisioning and GPU resource allocation, allowing engineers to focus on model performance and developer velocity rather than cluster maintenance.
  • Nutanix Cloud Infrastructure (NCI): Software-defined platform running directly on self-owned hardware in a self-managed datacenter, integrating compute, storage, and networking into a single unified foundation. This allows a seamless path to integration with existing IT processes and tooling, enabling operation and scale of private AI workloads using the same consistent operations and management workflows they rely on across their existing infrastructure.
  • Nutanix Unified Storage (NUS): Provides data services through high performance file and object storage, optimized for hosting and serving on-prem, open-weight AI models leveraged by Nutanix Private Inference, while providing operational metrics and system observability.
  • Nutanix Database Service (NDB): Provides a managed PostgreSQL instance in each environment to store Nutanix Enterprise AI data, including application state, model definitions, endpoint configurations, and access control information.
     
• Scalable AI Infrastructure Operations: Enables IT to run, scale, and manage AI workloads with resiliency, reliability, and operational control across the full infrastructure stack.
• Ensure Data Sovereignty: Hosting open-weight models on Nutanix owned infrastructure keeps proprietary code, IP, and sensitive data within Nutanix’s own environment. Supporting capabilities including role-based access control and governed, audited agent access to tools and data provide control and visibility.

“Building a private AI factory isn’t just about buying GPUs. The AI Factory is a full stack solution,” said Inbar Stolberg, Senior Director, Engineering at Nutanix. “It’s about creating a platform that can give our engineers the tools they need without the public cloud price tag. For us, that means delivering a lower cost per inference than commercial services, with strict governance, resiliency, and reliability on the new AI clusters.”

Nutanix Agentic AI Factory Architecture

Open-Weight Models running on the AI Factory include:

• Laguna S 2.1 (Poolside)
• NVIDIA Nemotron 3 Super (NVIDIA)
• Muse Glimmer (Meta)
• Muse Spark (Meta)
• Beam (Reflection)

Additional models are also planned to be deployed soon, including Thinking Machines’ Inkling model. Larger models (e.g. Laguna, Beam) are run across multiple GPUs to optimize performance and throughput, leveraging techniques such as tensor parallelism and KV cache aware routing. The Agent Gateway also routes to frontier models like GPT-5.6-Luna (OpenAI) via Amazon Bedrock.

The Customer Playbook for Enterprise AI

As AI adoption expands across the software development lifecycle, organizations face growing pressure to manage rising token costs while protecting sensitive data. For Nutanix, the AI Factory is both an investment in its own software development productivity and an opportunity to demonstrate how enterprises can manage AI costs and support data sovereignty objectives. 

Nutanix’s approach provides one example of how an organization can address these challenges by targeting an 80/20 split: running 80% of everyday agentic coding tasks locally on open-weight models, while the remaining 20% of high-end reasoning tasks get escalated to frontier cloud models. This operational model can be extended beyond AI coding to other core enterprise workloads, from everyday cowork productivity and internal search and data querying to real-time security threat triage. By combining owned GPU infrastructure, open-weight models, and access to frontier models, Nutanix’s internal deployment illustrates one possible framework that gives IT and business leaders control over AI costs and operational scale, while giving engineers and developers the flexibility to innovate.

Ready to Control Your AI Costs and Data?

Learn how Nutanix Agentic AI can help organizations like yours gain greater control over their AI costs and data sovereignty.