Understanding CPU Resource Management in Nutanix AHV

Introduction

The Nutanix AHV hypervisor uses hardware acceleration provided by the physical processor so that VMs can run CPU instructions as if they were running directly on the physical processor.

Similar to how many applications can run under one operating system simultaneously, AHV allows running multiple virtual machines in parallel.

The combination of the hardware acceleration and ability to run multiple virtual machines in parallel significantly increases the ability to maximize utilization of the available CPU capacity as most workloads will not be able to fully utilize their allocated CPU (for example, while they wait for other services, for user inputs, or for a response from I/O devices).

AHV virtualization is backed by a few key components, including KVM, which is part of the AHV kernel and provides access to this hardware acceleration and enables highly efficient running of virtual machines.

CPU Scheduling

AHV represents each vCPU core as a Linux process thread, and uses a process scheduler to individually schedule each vCPU core from a virtual machine onto an available physical thread of a core. The AHV scheduler distributes vCPU cores to physical threads based on the amount of CPU time they have already used, ensuring that there is fair balancing of the available CPU resources between all user virtual machines and supporting infrastructure.

The distributing nature of the AHV scheduler ensures that, if there is CPU oversubscription and there is more demand for physical processor resources than can be provided, the resources are shared between the virtual machines and services that require the CPU resources.

As this scheduling is based on individual vCPU cores, the actual resources that a VM receives will also be based on the number of vCPU cores that have been allocated to the guest.

In a highly contended system, where all virtual machines are asking for more processor capacity than is available, a VM with 8 vCPU cores will receive approximately 8 times as much processing time as a VM with just one vCPU core.

In such a system, the impact of not having sufficient processor capacity will be felt across all VMs proportionally to the number of vCPU cores that they have been allocated.

AHV doesn’t limit the amount of oversubscription that can be achieved on a host, however  Nutanix’s best practice for CPU oversubscription is that business-critical or latency-sensitive workloads should look to use either no over-subscription or up to a 2x over-subscription, while non-critical applications or those not latency-sensitive may use up to 4x over-subscription in the number of vCPU cores allocated. Note that many workload demands for CPU utilization will vary over time (e.g., backup tasks or security scans which may occur at specific times), so it’s worth considering this when determining how much over-subscription is appropriate for each use case.

Base Utilization

Unlike some other resources (like memory), processor capacity is not only related to the number of physical cores available but also to the processor generation and the processor clock speed. The processor’s clock speed is an indication of the relative performance of each core, so a processor with a higher clock speed but the same number of cores would have higher throughput. In addition, newer processor generations typically improve workload performance by running the same instructions faster, but also introduce new instructions that can be used to improve performance and efficiency even further.

For example, between a pair of processor generations with the same number of cores and clock speed, newer generations can show a 10%+ increase in one benchmark for Integer performance (SPEC CPU2017). Even when using Nutanix’s Advanced Processor Compatibility to control the CPU features presented to the VM, the performance benefits of increased clock speed, number of cores, and efficiency in the newer processor generations will all make workloads run more efficiently.

On a sample host with two Intel(R) Xeon(R) Silver 4214 CPUs and no CVM running, AHV itself uses less than 1% of the CPU resources. The base utilization without workloads running is low. This system has a total of 24 cores, which, with hyperthreading, make a total of 48 logical threads - so less than a quarter of a single core (or less than half a logical thread) is consumed by the base utilization of the system.

CVM Utilization

In Nutanix’s hyper-converged offerings, the services providing the software-defined element of the hyperconverged stack, notably the Controller Virtual Machine (CVM), are exceptions to the fair balanced distribution of processor resources. The CVM has multiple responsibilities, but two key ones are cluster and storage management.

Both of these are critical services - for example, cluster management services are needed to resolve CPU hotspots, and storage provided by the CVM is critical for keeping virtual machines running.

In order to ensure that the CVM is provided with adequate resources for these critical services, the AHV scheduler uses a weight-based system for the CVM ensuring it has higher priority (4 times more than user virtual machines) and is able to run harder and longer in contended scenarios. This is designed to maintain uptime and reliability. Outside of contended scenarios, the CVM is simply another VM being scheduled alongside all other workloads.

However, it may be difficult for the CVM to consume 4 times more resources than user virtual machines as for most systems it is also restricted by the number of allocated vCPU cores. In many systems the default number of vCPU cores allocated to the CVM would normally be 8 or 12, but may be as low as 6 for some environments. This allocation logic is built into the initial imaging of the host based on the hardware configuration profile detected by the Nutanix Foundation deployment tool .

The configuration sizing recommendations for the CVM include the recommended number of vCPU cores assigned to the CVM - but that doesn’t mean those cores aren’t able to be used by virtual machines. All cores on a system can be used by virtual machines, based on the fair scheduler and weighting. On the sample idle host as above, the CVM can use less than 3% of the CPU resources when not under load, meaning that the majority of the CPU resources for the host are available for workload-related activities (which could include network and storage needs, as described in the next section).

In total, in the 24-core (48-thread) system described above, just over 1 core (less than 5%) could be considered as the base utilization of the host plus CVM before virtual machines are started. But that's specific to this system - as you increase the number of cores, the clock speed, or the processor generation, the percentage used by the host plus CVM will decrease because while the capability is increasing, the work needed to be done is not changing.

Workload-dependent Utilization

The base utilization described above is just one factor in the total CPU utilization needed by the system. CPU utilization by the shared services increases when VMs start running on the system, consuming storage and networking services.

Nutanix HCI provides storage and networking services for the infrastructure and VMs. The storage services are provided through the CVM, and the network services are provided through the AHV host.

Like any environment which provides software-defined infrastructure, the amount of CPU utilization needed to provide these services will vary substantially based on the workloads that are running in the guests and on the wider cluster. In addition to the clockspeed and CPU generation discussed above, the CPU utilization for storage and networking will depend on a number of factors, including the following:

• Number of I/O operations being performed on the storage
• Sizes of the I/O operations being performed on the storage
• Amount and type of network traffic related to the virtual machines
• Types and number of microsegmentation rules
• Hardware offloads available

It’s worth noting that the CVM is providing services not just to the local host, but also to other hosts in the cluster. Nutanix’s clustered storage will replicate data to other hosts in the same cluster, with that data processed by the CVM on the remote host. As such, the CVM’s utilization of CPU resources will depend significantly on the utilization of data across the cluster. In addition to the impacts to CPU utilization from storage usage across the cluster, a similar impact could be seen in the network - an increase in broadcast, multicast or UDP traffic received is one example where additional processing would be needed on the host, which is not caused by running VMs or other services.

While the amount of CPU resources which the CVM consumes to provide storage services will vary, an assumption used by the Nutanix Sizer tool is that approximately half of the vCPU cores assigned to the CVM might be utilized to provide the storage across the cluster.

For example, if the CVM is configured with 12 vCPU cores (following Nutanix’s recommended CVM sizing), then a rule-of-thumb may be to assume that on average 6 physical cores would not be available to run user workloads. A more accurate way to determine how much CPU utilization is required by the CVM is to run representative workloads within the environment and measure it directly using the metrics exposed.

Another important consideration is the security-related mitigations that may need to be applied. Which security mitigations are needed is entirely dependent on the generation of the processor, and the impact of those mitigations on CPU utilization is heavily dependent on the tasks that a workload is performing. With older processor generations the impact could be quite significant, with additional CPU utilization needed to keep workloads secure, while newer processor generations have hardware mitigations built in, so don’t need the software-implemented workarounds which would otherwise be increasing the CPU utilization.

Removing CPU hotspots

In addition to sizing considerations, the Acropolis Dynamic Scheduling (ADS) feature (one of the cluster management services noted above as running in the CVM) is continually monitoring nodes in the cluster to ensure that we do not create CPU hotspots that may impact the running of VMs when there is capacity elsewhere in the cluster.

CPU utilization is considered to have created a hotspot when the total CPU usage on the host exceeds 85%. This host utilization includes any processes running by the host itself, including those servicing networking, and the VMs’ usage.

When calculating which VMs to move to mitigate hotspots, ADS will also separately account for the storage CPU usage from within the CVM, increasing a VMs’ effective CPU utilization by the portion of storage CPU utilization that can be attributed to that VM’s disks. This ensures that ADS can accurately model the impact to a potential target host based on the VM’s total utilization.

In effect, with ADS mitigating CPU and storage hotspots, individual hosts in the cluster should not exceed 85% utilization of sustained workload.

CPU Resource Utilization - Summary

In general, the amount of CPU resources utilized in support of virtual machines depends on many factors, which would all need to be considered to make predictions of the utilization. The factors that impact the amount of CPU resources consumed in a system include:

• Hardware setup, including the CPU generation, clock speed, core count and any hardware-offloads configured
• Infrastructure configuration, including for features such as Flow Network Security and CVM sizing
• Virtual machine workload, not just CPU utilization, but also storage and network utilization sizes and patterns

In many environments, following the rule-of-thumb described above (assuming that half of the cores assigned to the CVM should be reserved to not run workloads) will give a good capacity estimate, but in some environments with particular workload or storage patterns, measuring the level of CPU utilization in the CVM may show either more or less utilization than this rule of thumb suggests.

The use of a representative external benchmark (such as SPEC CPU2017) to compare the relative performance of an older and newer processor can help scale existing workloads. If an existing cluster is running at 50% CPU utilization, and the representative benchmark suggests that the CPU selected for the new cluster is twice as effective at running the same task, then it may be fair to assume that the existing workloads would only see a 25% CPU utilization on the new cluster.

To understand the sizings for both the rule-of-thumb for CVM usage and the relative workload performance between CPU generations, check out Nutanix Sizer, which is a web-based tool designed to help organizations plan and size their Nutanix infrastructure for various workloads where these factors are part of the implementation.

Measuring CPU utilization to understand usage patterns and estimate future usage can become very cumbersome, so to manage your estate after the initial deployment, Nutanix offers capacity planning tools as part of the NCM Intelligent Operations solution. These tools help track capacity of the estate, make predictions and run “What If” scenarios to make capacity planning seamless.

©2025 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and all Nutanix product and service names mentioned herein are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. All other brand names mentioned herein are for identification purposes only and may be the trademarks of their respective holder(s).

Our decision to link to or reference an external site should not be considered an endorsement of any content on such a site. Certain information contained in this post may relate to, or be based on, studies, publications, surveys and other data obtained from third-party sources and our own internal estimates and research. While we believe these third-party studies, publications, surveys and other data are reliable as of the date of this paper, they have not independently verified unless specifically stated, and we make no representation as to the adequacy, fairness, accuracy, or completeness of any information obtained from a third-party.

All code samples are unofficial, are unsupported and will require extensive modification before use in a production environment. This content may reflect an experiment in a test environment. Results, benefits, savings, or other outcomes described depend on a variety of factors including use case, individual requirements, and operating environments, and this publication should not be construed as a promise or obligation to deliver specific outcomes.

This content may reflect an experiment in a test environment. Results, benefits, savings, or other outcomes described depend on a variety of factors including use case, individual requirements, and operating environments, and this publication should not be construed as a promise or obligation to deliver specific outcomes.