Provisioning MongoDB Sharded Clusters on Nutanix Database Service

How we redesigned NDB's provisioning engine, and the three engineering decisions that help make large-scale MongoDB sharded clusters fast, predictable, and secure.

As more enterprise customers started running MongoDB at scale, a clear pattern emerged: deployments weren't limited to just 2 or 3 shards anymore. Requests for 10+ shard configurations were becoming the norm. That's not just a bigger cluster, it's a fundamentally different orchestration problem.

A MongoDB sharded cluster is a hierarchy of independent components. Each shard is its own replica set. The config server is its own replica set. Mongo's routers must be bootstrapped and wired to the topology. The moment you treat this as n times the work of a single replica set, provisioning time becomes a wall that grows linearly with cluster size. At 10 shards and 44 VMs, that's not acceptable.

This post walks through how we approached this problem in NDB, the data model we built, how we designed the provisioning pipeline for parallelism, and what the results look like at scale.

Result: Based on internal development testing - 4.4× increase in cluster size (10 → 44 VMs) produced only a 1.7× increase in provisioning time. A 44-VM sharded cluster provisions in under one hour.

What is Nutanix Database Service?

Nutanix Database Service (NDB) is a Database-as-a-Service platform that automates the full lifecycle of enterprise databases, including provisioning, cloning, patching, backup, and recovery across hybrid cloud environments. It supports Oracle, SQL Server, PostgreSQL, MySQL and MongoDB through a single control plane, turning multi-step DBA workflows into automated operations. The sharded cluster work described here is available starting with NDB 2.10.

1. Getting the Data Model Right

Before we could parallelize provisioning, we had to solve a modeling problem. NDB already had a well-established model for databases: one logical database entity, one Time Machine for point-in-time recovery and one set of VMs forming a replica set. It worked well for standalone instances and replica sets.

A sharded cluster breaks that model. You have multiple replica sets - one per shard, plus the config server - each with its own VMs and state. The question was whether to build a separate one-off path for sharded clusters, or extend the existing model in a way that would be maintainable and consistent.

We chose to extend. The design principle was simple: if we model each constituent replica set as a first-class entity in NDB, we get two things for free: all the existing NDB machinery (Time Machine, backup, patching) works per-shard without modification and adding or removing a shard becomes a clean attach/detach operation rather than a mutation of a monolithic structure.

What This Unlocks

Shard scale-out without disruption: When a customer needs to add a shard, NDB provisions the new logical cluster (inc. VMs, storage, MongoDB binaries) completely independently of the running cluster. Only once a new shard is fully healthy and validated does NDB register it with the Parent Database and trigger MongoDB's balancer to begin chunk migration. The production cluster is untouched during the build phase. Removing a shard follows the reverse process : detach from the parent first and decommission infrastructure in the background.

Unified backup across all shards: A single snapshot operation against the Parent Database is broken down into parallel snapshot jobs - one per logical cluster - executing simultaneously. Backing up 10 shards takes roughly the same wall-clock time as backing up one, bounded only by storage I/O. No manual per-shard coordination.

2. Parallel-First VM Creation

For improving sharded cluster provisioning performance, one of the key areas of evolution was how the underlying virtual machines are created and configured. Earlier, NDB relied on Offline Disk Processing (ODP), where the VM’s OS disk was prepared before boot by mounting it to a helper VM and injecting the required configuration. This approach provided a reliable and controlled way to ensure consistent system setup. However, as cluster sizes increased, especially for large sharded deployments, the sequential nature of preparing disks one at a time introduced a natural throughput limit.

To enable faster provisioning at scale, the architecture evolved toward a model that supports parallel initialization. NDB now integrates cloud-init, a widely used cloud instance initialization framework that allows a VM to automatically configure itself at first boot using metadata provided by the hypervisor.

How It Works

Our design goal was to have the provisioning engine initiate the creation of all cluster VMs in a single pass, while each node independently handles its own configuration. Cloud-init makes this possible. Rather than preparing each VM's configuration before boot, Cloud-init shifts configuration to boot time. Network settings, SSH keys, and hostname are passed to the hypervisor as user-data metadata at the point of VM creation. Every VM boots simultaneously and the cloud-init service inside each guest OS applies its configuration concurrently. The provisioning engine issues one creation wave for all VMs, including those for the Config Servers, Shards and Mongos routers, and then moves on.

Infrastructure note: The control plane of NDB can issue creation requests for all VMs in a single pass. Effective concurrency is bounded by the underlying hypervisor (AHV or ESXi) and storage fabric capacity - not by the orchestration logic.

3. Parallelism Through the Stack: Storage and Services

Once VMs are online, the parallel execution continues up into storage attachment and MongoDB service initialization.

Storage Attachment: Distributed Execution

Shard nodes and config servers all require persistent storage. NDB targets the entire group simultaneously; every node initiates iSCSI discovery and volume attachment in parallel across the Distributed Storage Fabric of Nutanix. There is no inter-shard dependency here. Shard 1 formatting its data disk has no bearing on when Shard 2 begins.

Service Initialization: Multi-Threaded Orchestration

MongoDB service initialization follows the same principle. The NDB control plane spawns an independent thread per shard. Each thread drives its logical cluster through the full sequence MongoDB service start and replica set formation without waiting on any other shard. The initialization pipelines start concurrently, with Config Server initialized first, then followed with Shard A, Shard B, ... , Shard N pipelines.

Having a single synchronization point is intentional - once all replica sets are initialized, they are bound to the mongos routers and the cluster topology is assembled. This final step involves metadata coordination across all shards and the config server. It scales with cluster size and that's expected. This is the inherent cost of a larger cluster, not a pipeline inefficiency.

Sharded Cluster Registration with NDB

After the sharded cluster becomes operational, the final phase involves registering cluster components with the NDB control plane. This process is also executed in parallel. Each shard registers its logical cluster independently with NDB, persisting the required metadata in the NDB repository. This includes information such as disk layouts for snapshot management, replica set topology, and the identification of primary and secondary nodes. The Config Server registers first, after which all shard logical clusters register concurrently. There is no dependency between shards during this stage. Similarly, mongos routers also register with NDB in parallel, allowing the control plane to build a complete view of the cluster topology without introducing serialization into the workflow.

Diagram of the NDB provisioning pipeline for MongoDB sharded clusters: parallel VM creation with cloud-init, parallel storage attachment, parallel MongoDB service configuration, and sharded cluster registration with NDB

4. Encryption at Rest: TDE at Provisioning Time

Provisioning performance is one dimension. Security posture is another, and for enterprise deployments they need to be addressed together. One key security capability is Transparent Data Encryption (TDE), which protects data at rest by encrypting the database files on disk so that even if storage media is accessed directly, the underlying data remains unreadable without the encryption keys.

Why Provisioning Time Is the Only Right Time

For MongoDB, encryption at rest must be enabled at node initialization; it is baked into how WiredTiger initializes its data files. There is no in-place re-encryption process. If encryption is not enabled at provisioning time, the only path to enable it later is to take each node offline, wipe its data files, and resync it from scratch with encryption enabled from initialization.

For a simple replica set, this is a high-risk rolling operation that reduces redundancy and leaves the cluster vulnerable if the primary fails mid-process. For a sharded cluster, with multiple replica sets, a config server replica set, and mongos routers, this is not practically feasible on a live production system. It is effectively a full cluster rebuild.

In practice, enterprises that miss encryption at provisioning time are stuck. The remediation cost is high enough that most teams either accept the risk or plan a full re-deployment.

We integrated TDE into the provisioning workflow in NDB 2.10, so clusters come up encrypted from the start.

How TDE Works in NDB

TDE operates at the WiredTiger storage engine level, MongoDB's native storage layer. Encryption is applied to data files before they are written to disk and decrypted transparently on read. Applications require no changes and see no performance difference in normal operation.

Key management is handled through KMIP integration. During provisioning, NDB connects to a KMIP-compliant key management server, retrieves or generates encryption keys, and configures each node's WiredTiger instance to use them. The keys themselves never reside on the database nodes.

Why This Matters Operationally

For organizations with data-at-rest encryption requirements, such as PCI-DSS, HIPAA, SOC 2, or internal security mandates, the traditional risk has been missing encryption at provisioning time and facing an effectively irreversible situation on a live production cluster. NDB is designed to avoid that risk. NDB supports customers’ compliance efforts from day one, with no re-encryption window, no post-deployment remediation, and no rebuild risk.

5. Does It Actually Scale Sub-Linearly?

Our design goal was sub-linear provisioning time, which meant that the cluster size would grow faster than the provisioning time. We validated this through controlled QA benchmarking on identical infrastructure, varying only cluster topology.

What We Expected

If parallelism is working correctly across VM creation, storage attachment, and service initialization, then doubling the number of nodes should not double provisioning time. The only phase expected to affect scalability is cluster registration, where metadata coordination across shards and config servers is inherently sequential by nature.

Test Setup

  • Small Cluster (10 VMs): 2 shard replica sets (3 nodes each), 1 config server replica set (3 nodes), and 1 mongos router ; Total Operation Time: 30 mins
  • Medium Cluster (20 VMs): 5 shard replica sets (3 nodes each – 15 VMs), 1 config server replica set (3 nodes), and 2 mongos routers ; Total Operation Time: 37 mins
  • Large Cluster (30 VMs): 8 shard replica sets (3 nodes each – 24 VMs), 1 config server replica set (3 nodes), and 3 mongos routers; Total Operation Time: 43 mins
  • Huge Cluster (44 VMs): Mixed configuration with 5 shard replica sets (3 nodes each) and 5 shard replica sets (4 nodes each) totaling 35 VMs, 1 config server replica set (5 nodes), and 4 mongos routers; Total Operation Time: 51 mins

Please note that all the tests were executed on the same NDB server, with identical hardware, cluster setup, CPU, memory allocation, and data size across all database nodes on a setup where nodes are spread across three different Nutanix physical clusters.

Results

~30 min
10 VMs
~51 min
44 VMs

A 4.4x increase in VMs produced only a 1.7x increase in provisioning time. The full 44-VM cluster is provisioned in under one hour.

Stacked bar chart of provisioning time by phase for 10, 20, 30 and 44 VMs, with total time rising from about 30 to about 50 minutes

Breaking Down the Phases

  • VM creation: Scales in parallel via Cloud-Init. Additional nodes add negligible wall-clock time up to hypervisor concurrency limits.
  • Storage attachment: Distributed across all nodes simultaneously. No inter-shard dependency.
  • Service initialization: Multi-threaded per shard. Bounded by the slowest individual shard, not by total count.
  • Cluster registration: Scales with cluster size this is the expected cost of coordinating a larger topology. It shows up in the data and is accounted for in the 1.7x figure.

Conclusion

The redesign transforms MongoDB sharded cluster provisioning from a sequential process into a fully parallel workflow and the three design decisions above drove this result. First, modeling the sharded cluster correctly extends the existing replica set model of NDB so each shard is a first-class logical cluster, which in turn made parallelism structurally possible and made shard scale-out a clean attach/detach operation. Second, designing VM creation for parallel execution from the start meant node count doesn't drive provisioning time linearly. Third, integrating TDE at provisioning time means security isn't a post-deployment concern.

The benchmark result of 44 VMs in 51 minutes reflects all three design decisions working together. For teams running MongoDB at enterprise scale, it means large sharded cluster deployments are a routine, predictable operation.

©2026 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and all Nutanix product and service names mentioned are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. All other brand names mentioned are for identification purposes only and may be the trademarks of their respective holder(s). This content reflects an experiment in a test environment. Results, benefits, savings, or other outcomes described depend on a variety of factors including use case, individual requirements, and operating environments, and this publication should not be construed as a promise or obligation to deliver specific outcomes.