Introduction
AI training and inference workloads require high throughput to maximize performance. The Nutanix Unified Storage (NUS) Elite solution provides this capability by implementing single root I/O virtualization (SR-IOV) passthrough network interfaces and a dedicated Remote Direct Memory Access (RDMA) high-speed data path. This design allows object storage performance to scale linearly with physical network speeds and is designed to deliver high-efficiency data for demanding AI and machine learning workloads. By bypassing the legacy kernel network stack, this approach can achieve near line-rate saturation. In internal testing, this design reduced client CPU overhead by up to 98 percent during GET operations, reclaiming vital compute resources to maximize the performance and ROI of these workloads.
The Challenge: Objects Storage is Not Designed to Meet AI Performance Demands
The Nutanix Objects Storage solution traditionally relies on a VM providing an S3 endpoint to clients using an Open vSwitch (OVS) bound interface and a second OVS-bound interface making Remote Procedure Calls (RPCs) to Nutanix AOS to store the object data on the disks. In an all-flash world, this solution means performance is ultimately limited to whichever is less: your CPU capability or your network bandwidth.
Traditional deployments using OVS-bound interfaces might encounter performance constraints as throughput demands increase. This architecture can present challenges when attempting to fully use high-speed network bandwidth.
Additionally, in Nutanix Objects Storage versions prior to 5.4, connections were load-balanced through an Envoy instance. This Envoy instance operated within OVS networking limits, creating a potential for bottlenecks similar to those with the Object Controller (OC). Removing this bottleneck in Objects 5.4 provided significant performance improvements, but not enough to scale to 200 Gb networking and beyond.
The Solution: Nutanix Unified Storage Elite
We solved the OVS limitation by adding SR-IOV passthrough network interfaces to the Object Controller (OC), enabling us to scale performance with network speed. The hardware design is identical to the one Nutanix Files Storage used for NVIDIA Enterprise Certification.
Hardware Options
- NX-8170 Node:
- (2) Intel 32 Core CPUs (64 Cores total)
- 1TiB Memory
- (12) 15.36TiB NVMe SSD
- (2) Mellanox CX7 Network Adapters
- NX-8150 Node:
- (2) 32 Core CPUs (64 Cores total)
- 1TiB Memory
- (20) 15.36TiB NVMe SSD
- (3) Mellanox CX7 Network Adapters
Software Stack
- Objects Storage 5.4 (Front-end S3)
- AOS Storage 7.6 (Storage and control plane)
- Prism Central 7.6 (Management)
- Nutanix AHV 11.2 (Hypervisor)
In this setup, we configure a full NUMA node for the Controller VM (CVM) and another for the OC, enabling us to use the system’s full CPU capability and memory bandwidth. We additionally segment network interfaces to avoid oversubscribing the network where possible. The following is a sample wiring diagram:
Fig01. Network Connections for Single Objects Storage Node (NX-8170)
Migrating the OC network interfaces to SR-IOV passthrough NICs also fundamentally changes how the OC interacts with the CVM to access block storage. In this updated design, one of the passthrough NIC’s on the OC is dedicated to client access, while the other passes data between the OC and CVM (using the CVM’s rdma0 interface). The high-speed data path bypasses the standard OS network stack entirely. With RDMA, we can maximize throughput for back-end storage communication.
Fig 02. Overview of SR-IOV Passthrough Network Interfaces and RDMA High-Speed Data Path
Performance of the High-Speed Data Path
In benchmarking a four-node cluster with our high-performance design, we scaled to approximately 85 percent of port bandwidth saturation for GET requests and 36 percent for PUT requests across various object sizes.
Fig 03. S3/TCP Throughput by Object Size: 8 Clients, 4 Object Nodes, 64 Threads
Additional Improvements with RDMA
With the cuObject Server library integrated directly into the Object Controller as of Objects 5.4, S3 requests gain transport-layer enhancement through S3/RDMA. This integration marks a key advancement toward delivering native GPU direct storage support for S3. The benefits of RDMA are well known in the storage world, and we saw expected results in comparing S3/TCP with S3/RDMA.
Fig 04. GET Bandwidth Throughput (MiB/s): 8 Clients, 4 Objects Nodes, 64 Threads
Fig 05. PUT Bandwidth Throughput (MiB/s): 8 Clients, 4 Objects Nodes, 64 Threads
For most object sizes, GET requests with RDMA were substantially faster, enabling more than 90 percent network bandwidth utilization on our four-node cluster. The PUT performance reached parity between TCP and RDMA. For smaller object sizes, TCP prevailed due to long-standing enhancements in session handling.
In addition to increased throughput, RDMA transport provides benefits by returning resources to your applications.
Fig 06. Achieved Bandwidth vs. CPU Cost: 8 Clients, 4 Objects Nodes, 64 Threads
With TCP transport, each GB/s of storage bandwidth costs at least 1 full client CPU core. Here, RDMA significantly reduces the overhead applications incur when accessing Object Storage.
Conclusion and Takeaways
The transition from S3/TCP to S3/RDMA with the Nutanix Unified Storage Elite framework (Early Access) represents a transformative leap in object storage capabilities. By moving beyond traditional OVS networking constraints, this architecture allows storage performance to scale linearly with physical network capacity, achieving near line-rate saturation and allowing CPU resources to be focused on applications rather than infrastructure overhead.
- Performance gains: S3/RDMA facilitates cycle reclamation, reducing CPU utilization by approximately 98 percent during GET operations in our internal testing. This reclamation, paired with significant throughput increases, provides the high-performance data patterns necessary for demanding AI and machine learning workloads.
- Architectural shift: Implementing SR-IOV passthrough NICs and an RDMA-based high-speed data path allows the system to bypass the legacy kernel network stack. By design, this decoupling provides that storage throughput is no longer gated by host CPU interrupts or software bottlenecks.
©2026 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and all Nutanix product and service names mentioned are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. All other brand names mentioned are for identification purposes only and may be the trademarks of their respective holder(s). This content reflects an experiment in a test environment. Results, benefits, savings, or other outcomes described depend on a variety of factors including use case, individual requirements, and operating environments, and this publication should not be construed as a promise or obligation to deliver specific outcomes.
This content may contain express and implied forward-looking statements, including related to the Nutanix Unified Storage (NUS) Elite early access release (not GA) solution, which are not historical facts and are instead based on our current expectations, estimates, and beliefs. The accuracy of such statements involves risks and uncertainties and depends upon future events, including those that may be beyond our control, and actual results may differ materially and adversely from those anticipated or implied by such statements. Any forward-looking statements included speak only as of the date hereof and, except as required by law, we assume no obligation to update or otherwise revise any such forward-looking statements to reflect subsequent events or circumstances.
