Introduction

Ever wondered how your Nutanix AHV cluster keeps running smoothly without any one host getting overloaded? Or how VMs are rearranged in a cluster to make room for new VM when no host has enough free resources Or how your VM policies like VM-VM anti-affinity, VM-Host affinity, and HA reservations are honored in the cluster.

That’s where the Acropolis Dynamic Scheduler (ADS) solution comes in - it's Nutanix’s intelligent, built-in scheduler designed specifically for AHV.

ADS proactively monitors the cluster to address resource contentions and optimize VM placement. Operating at a cluster level, ADS continuously monitors resources across all of the hosts in the cluster to deliver efficient workload distribution. 

As ADS focuses on resource contention remediation, VMs are migrated only when necessary, compared to the frequent migrations typically needed for an active load balancing approach, which can negatively impact VM performance.

In addition to resource optimization, ADS performs several key operations, including enforcing VM policies such as VM-VM anti-affinity, VM-Host affinity, providing High Availability (HA) reservations, and balancing iSCSI sessions from external targets across CVMs (Controller VM) in the cluster for Nutanix Volume Groups.

How is resource contention remediated?

ADS proactively monitors resource usage of all VMs on the Host to detect and resolve any resource contention. Contention is defined as “Host CPU utilization” or “CVM storage controller CPU utilization” exceeding 85%, or VMs memory swapping to disk on the Host when memory overcommit is enabled on VMs.

ADS runs every 15 minutes to detect and resolve Host CPU or CVM storage controller CPU contention, and every 2 minutes to detect and resolve memory overcommit contention.

Proactive Monitoring: ADS monitors VM CPU utilization, CVM storage controller CPU usage per VM, and memory utilization (including disk swapping) of VM when memory overcommit is enabled on the VM. It also monitors iSCSI target usage in the case of volume groups.

Resource Contention Detection: To detect CPU and CVM storage controller CPU contention on the Host, ADS considers CPU and CVM storage controller CPU usage of all the VMs on the Host and monitors the trends of the data in the last 10 minutes. This trend analysis filters out transient spikes, such as those from VM boot storms, which do not indicate persistent resource contention. For memory overcommit contention detection, it considers the VMs memory utilization and observes for disk swap every 2 minutes.

Resource Contention Remediation: when resource contention of Host CPU and CVM storage controller CPU is detected, ADS creates a migration plan to redistribute VMs across the cluster to remediate resource contention i.e. keep resource utilization of all nodes within 85%. In case of memory contention, ADS will attempt to resize the memory pool of the overcommit enabled VMs so that more physical memory can be allocated to VMs to avoid disk swapping. If there is insufficient physical memory available on the host to prevent VMs from using swap, ADS attempts to create a migration plan every 15mins to redistribute the VMs across the cluster so that VMs get allocated sufficient physical memory. While VMs are migrated as part of resource contention remediation plan, if the VM has vGPU profile configured, it will migrate it to the Host which has a compatible GPU card.

When there are multiple resource contentions in the cluster, ADS attempts to create a migration plan to resolve all and if it cannot resolve all the resource contentions with policy compliance, it refrains from creating migration plans.

After completing a migration plan, ADS will monitor the cluster for a short period to ascertain the impact of the migration plan before initiating another migration plan. 

Migration Cost and Node Scoring: ADS creates the migration plan with the least migration cost to minimize the impact from the VM migrations. It considers VM Migration cost and Host node score to create the migration plan.

Migration cost of the VM is calculated based on the VMs configured memory and GPU resources attached. In case of memory overcommit enabled VMs, memory usage including swap is also considered.

Similarly, the Host node score is calculated based on the utilization of the Host, i.e., CPU, Memory, and GPU. Higher utilizations result in higher scores.

ADS prioritizes reducing VMs combined migration cost over increasing load on the hosts as long as the migration doesn’t lead to resource contention. When multiple destination nodes are viable, it prefers those with lower node scores.

Example resource contention remediation

Consider a 3-node cluster with the below node and VM utilization.

ADS Migration Plan to resolve resource contention

To resolve the above resource contention, ADS formulates a migration plan as shown below by first freeing up space on Node 3. It does this by migrating a VM with 15% CPU utilization from Node 3 to Node 2. This creates enough capacity on Node 3 for ADS to migrate a 45% CPU utilizing VM from Node 1 to Node 3. This migration plan remediates resource contention, so that all Nodes stay within the 85% utilization threshold. Refer to resource utilization of all nodes post the migrations.

How does ADS assist with Initial VM Placement?

During VM creation, if there are no sufficient resources on any of the nodes in the cluster or if the nodes do not meet the VM policy requirements, ADS will attempt to rearrange existing VMs within the cluster to create the necessary space on one of the nodes and meet policy compliance for the new VM. As part of the redistribution attempt, if the plan would lead to resource contention, ADS will reject that VM deployment.

In cases where the VM is configured with a vGPU and no compatible GPU card is available, ADS will reorganize VMs across the cluster to free up a suitable GPU card. This allows the freed up GPU card to be partitioned according to the required profile, enabling the successful placement of the new VM.

Policy Enforcement and High Availability Reservation

ADS considers VM-Host Affinity, VM-VM Anti-Affinity policy and High Availability reservation during initial VM placement and resource contention remediation migrations. In case of initial VM placement, VM-VM Anti-Affinity policy is soft enforced, as starting the VM is preferred over non-compliance of VM-VM Anti-Affinity rules. However, the Host-Affinity policy is always strictly enforced. In case of VM migrations as part of the resource contention resolution plans, if it would result in the violation of these policies, ADS will reject the migrations to maintain policy compliance.

ADS is also responsible for High Availability reservation and restarting the VMs during HA events. For Guaranteed High Availability configuration, ADS confirms there is enough space within the cluster to restart the VMs during HA events. Resources allocated for HA recovery will depend on whether reservation is configured for single or dual node failure.

Memory Overcommit Interaction

ADS manages the physical shareable memory among the memory overcommit enabled VMs on the Host to support initial VM placement and to allocate memory to the memory overcommit enabled VMs based on the VMs memory need estimated by the Host. ADS monitors the memory utilization of the memory overcommit enabled VMs every 2 minutes to assess the physical memory, and disk swap used (if any) to identify whether the VMs need more memory to increase physical shareable memory or they have more memory assigned to them than the actual memory requirement to balloon it when needed.

Refer to AHV Internals: Memory Overcommit for more details.

Migration Tasks Audit

All the ADS VM migrations tasks are captured in the Prism Central. To view the list, go to Prism Central→Infrastructure→Activity→Audits and check in the right side pane filter for Migrate tasks under Operation Type.

Summary

ADS proactively monitors the cluster to resolve resource contentions, optimize VM placements dynamically, provide VMs with the resources they need, and keep them compliant with their policies. ADS is enabled by default and it is recommended to leave it as such to leverage all the benefits mentioned above.

©2025 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and all Nutanix product and service names mentioned herein are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. All other brand names mentioned herein are for identification purposes only and may be the trademarks of their respective holder(s).

Code samples, snippets and/or methods that appear in this content are unofficial, are unsupported and will require extensive modification before use in a production environment. As such, the code samples, snippets and/or methods are provided AS IS and are not guaranteed to be complete, accurate, or up-to-date. Nutanix makes no representations or warranties of any kind, express or implied, as to the operation or content of the code samples, snippets, or methods. Nutanix expressly disclaims all other guarantees, warranties, conditions and representations of any kind, either express or implied, and whether arising under any statute, law, commercial use or otherwise, including implied warranties of merchantability, fitness for a particular purpose, title and non-infringement therein.

This content may reflect an experiment in a test environment. Results, benefits, savings, or other outcomes described depend on a variety of factors including use case, individual requirements, and operating environments, and this publication should not be construed as a promise or obligation to deliver specific outcomes.