Scaling MongoDB Sharded Clusters with Nutanix NDB

The Problem: Growth Meets Complexity

A MongoDB sharded cluster does exactly what it promises -- distributes data across multiple replica sets so your application can handle more writes, more reads, and larger datasets. But there's a moment in many sharded deployments' life when the existing topology isn't enough. Maybe your working set has exceeded MongoDB's recommended WiredTiger storage thresholds (typically 2 TB per shard). Or perhaps your write latency is climbing. Beyond these comfort zones, performance degrades: cache pressure increases, compaction cycles slow down, and write latency climbs. The answer is straightforward: add another shard.

In a standalone MongoDB environment, this is a sequence of rs.initiate(), addShard(), and some balancer tuning. NDB 2.11 introduces seamless, scale-out for MongoDB sharded clusters, allowing users to add new shards and mongos routers to a live, running, backup-protected cluster with zero impact on the underlying infrastructure. This post covers how this live scale-out works under the hood, the design decisions behind it, and how we keep the database protected throughout the entire operation.

What Adding a Shard Actually Involves

Horizontal scale-out allows data and workloads to spread across more independent replica sets, keeping individual nodes (defined as independent database server VMs running your database processes) operating within a healthy resource envelope. However, while adding a shard sounds conceptually simple, the operational reality is substantial.

Before a single MongoDB command runs, someone has to stand up compute, storage, and networking -- provision VMs, attach disks, and configure network reachability across the cluster. Then comes OS hardening, MongoDB installation and configuration, replica set initialization, and security artifact setup (TLS certificates, authentication keys, potentially Kerberos). Once the shard is running, it must be added to the cluster's topology. MongoDB's balancer can then redistribute chunks across the expanded cluster based on the user's balancer configuration and workload requirements.

That's just the database layer. If the cluster is protected by any backup tooling -- as it always should be in production -- the new shard must also be incorporated into the backup configuration. The next snapshot must be verified for cluster-wide consistency. Restore workflows should be validated against the updated cluster topology to restore future backups successfully. Oplog tailing (the process of reading MongoDB's operation log to capture real-time changes for incremental backups) configurations need updating. And all of this must happen in the right order, with the right coordination, or you risk a backup system that silently covers N-1 shards out of N -- a gap that surfaces only during a restore, the one moment you can least afford it.

When MongoDB Ops Manager is part of the stack, providing automation, monitoring, and third-party backup APIs -- the coordination requirements multiply further. Ops Manager's Automation Agent is the single source of truth for cluster topology; shard additions must go through its Automation Config API. Backup agents must be enabled on new nodes, and the backup daemon's internal job references must be refreshed to reflect the changed topology. Also, oplog tailing nodes must be reconfigured.

NDB absorbs all of this friction. Rather than requiring operators to manually coordinate infrastructure provisioning, MongoDB topology changes, Ops Manager API calls, backup agent activation, and Time Machine configuration in the correct sequence, NDB provides a single scale-out operation that handles the full stack -- from VM creation to the first post-scale-out backup.

How NDB does it

NDB's scale-out for MongoDB sharded clusters is a two-phase operation: Provision, then Add.

Provision creates the infrastructure -- VMs, disks, networking, OS configuration, MongoDB installation, and replica set initialization. The new shard also inherits the existing cluster configuration, including security settings such as WiredTiger Encryption at Rest (TDE), when enabled, resulting in consistent behavior across the cluster. At this stage, the shard is registered with a forward reference to its target sharded cluster, but it remains in a “pending” state. It appears in topology views but is operationally inert: no backups run, and no snapshots are taken for it yet.

Between Provision and Add phases, you can install any required security artifacts on the new shard nodes -- such as x509 CA certificates for inter-node secure cluster communication or organization-specific TLS configurations. This separation introduces a deliberate trust boundary in the design. Automating certificate or keytab issuance would require NDB to hold persistent, privileged credentials to the customer's Certificate Authority (CA), HashiCorp Vault, or Active Directory (AD) -- effectively granting a database platform write access to core trust infrastructure. In many enterprises, the DBA or platform engineer using NDB doesn't own the CA or AD; that's a separate security or IAM team. The two-phase model respects this organizational boundary, keeping the provisioning automated while leaving credential issuance with the team that holds the proper authority.

Add orchestrates the shard's activation across NDB, Ops Manager, and the Nutanix storage layer in a single automated sequence:

1. Validation: NDB verifies that the new shard's IPs are whitelisted in Ops Manager, the Automation Agent is idle, and the balancer is not actively migrating chunks.
2. Agent Deployment: NDB installs the Ops Manager Automation Agent on each new shard VM or honors pre-installed agents based on the cluster's settings -- with dedicated log storage that matches the cluster's existing configuration.
3. Pause Protection: If backup protection is active, NDB pauses the cluster's hierarchical Time Machine and waits for any in-flight operations to complete. This prevents a snapshot from capturing a partially integrated shard.
4. Topology Update: NDB saves the current Ops Manager Automation Config as a rollback checkpoint, then submits the updated config with the new shard's replica set, monitoring it until the Automation Agent converges.
5. Backup Activation: NDB enables backup agents on the new shard nodes and refreshes Ops Manager's backup daemon with the expanded backup topology. Once Ops Manager recognizes the new shard as part of the protected cluster, backup services are resumed, including oplog tailing across all shards to maintain continuous incremental backup coverage.

Once everything is set, that is, the shard is activated in the cluster, and the backup topology is fully reconciled, Time Machine is restarted with the updated topology. A fresh baseline snapshot is automatically triggered to capture the full, updated cluster. While the end-to-end execution time varies based on the size and scale of your underlying data, NDB handles the entire sequence autonomously, turning what used to be hours of active DBA babysitting into a single 'fire-and-forget' reliable action.

Orchestrating Across Three Systems

To safely scale out a production cluster, NDB, the central conductor between the underlying Nutanix infrastructure, the MongoDB database engines, and MongoDB Ops Manager runs the operation safely. If any check fails along the way -- such as a busy balancer or an unreachable agent -- the system gracefully retries or halts before making destructive changes. The add-shard workflow coordinates NDB, Ops Manager, and the Nutanix infrastructure in a specific sequence:

Keeping Backups Consistent

The trickiest part of scale-out isn't adding the shard to MongoDB -- it's reconciling the architectural changes to keep backups consistent.

Topology-Aware Snapshots

Adding a shard changes the cluster's topology; that is, the number of replica sets increases, and MongoDB itself imposes a constraint here: a snapshot taken under the old topology cannot drive an in-place restore onto the new one. Restoring an N-shard snapshot onto an N+1-shard cluster would leave the config server metadata inconsistent, pointing at data that doesn't yet exist on the new shard. This is inherent to how MongoDB sharding works, not specific to NDB -- any backup tool sitting on top of a sharded cluster faces the same reality.

NDB handles this lifecycle complexity for you. After a successful shard addition, NDB automatically marks prior snapshots as non-restorable for in-place restores. Those snapshots are not deleted -- they remain available for cloning and other use cases -- but any direct restore attempt against them returns a clear, actionable error rather than silently producing an inconsistent cluster. The post-scale-out snapshot, triggered automatically when Time Machine resumes, becomes the new baseline for all subsequent restore and Point-in-Time Recovery (PITR) operations. The operator doesn't have to track any of this by hand.

Coordinating Time Machine During Scale-Out

During a shard addition, NDB pauses backup protection of the sharded cluster. This pause cascades across the cluster's Time Machine hierarchy. This prevents a race condition where a snapshot firing mid-addition could capture an incomplete, split-brain topology state, with some components reflecting the new shard while others do not.

Internally, NDB coordinates the hierarchical Time Machine model for sharded clusters: each shard has its own child Time Machine, and the sharded cluster has a parent that aggregates them. After the shard is successfully integrated and the backup topology is reconciled, NDB resumes Time Machine protection and triggers the post-scale-out snapshot to establish a clean PITR foundation.

Designing for Failure: The Rollback Trade-off

Scale-out spans multiple distributed systems, and partial failures are inevitable. The question isn't whether failures happen -- it's what state the database cluster is left in when they do.

We evaluated several rollback strategies and decided to fully rollback via Automation Config. Before submitting the expansion configuration payload to Ops Manager, NDB saves the original automation config as a rollback checkpoint. If any post-addition step fails, NDB restores the saved configuration, returning the cluster topology to its pre-addition state. If the rollback succeeds, the cluster returns to full operational health; otherwise, NDB surfaces the failure rather than leaving an ambiguous, partially integrated shard unnoticed. This layered strategy keeps the cluster functional in the common case and, in the worst case, prevents silent topology drift if recovery is required.

Mongos Scale-Out: The Simpler Sibling

Scaling query routers (mongos) is needed more frequently and urgently than scaling shards -- especially when read traffic spikes. Adding a mongos should be fast and low-risk.

NDB supports scaling both shards and mongos routers independently. Although new shards and mongos routers can be created in parallel, these should be added to the sharded cluster one at a time. This serialized approach keeps topology changes deterministic, simplifies validation, and enables backup metadata to stay consistent throughout the operation.

NDB applies the same familiar two-step pattern (Provision, then Add) for mongos routers. Like new shards, newly provisioned mongos routers inherit the existing cluster configuration, allowing them to integrate seamlessly into the deployment. But the operation stakes are considerably simpler due to their architecture:

• Stateless: A mongos instance holds no data, requires no Time Machine protection, and has no local snapshots to reconcile.
• No Topology Friction: Adding a mongos does not alter the underlying data shards, meaning it has zero impact on previous snapshot restore compatibility.
• Streamlined Recovery: If an NDB metadata update fails after Ops Manager successfully deploys the mongos, the operation can be safely retried or cleanly rolled back using the same Automation Config pattern but with lower stakes.

What This Means in Practice

• Zero-gap Backup Coverage: The new shard is fully protected by Time Machine -- snapshots, oplog backups, and PITR -- from the moment it becomes an active cluster member. There is no window where a shard exists in the cluster but is missing from backups.
• Consistent Restore: Topology-aware snapshot validation prevents operators from accidentally restoring a snapshot that doesn't match the current cluster layout. Restore operations across the full cluster remain coordinated and predictable.
• Operational Simplicity: What would otherwise require manual coordination across VM provisioning, MongoDB configuration, Ops Manager API calls, backup agent enablement, oplog node configuration, and Time Machine integration is now reduced to just two clean, simple NDB API calls (provision, then add).
• Safe Failure Handling: Partial failures during scale-out won't leave your cluster in an ambiguous state. Either the shard is fully integrated with backup protection, or the cluster is cleanly rolled back to its pre-scale-out state.
• Enterprise Security Compliance: The distinct two-phase model accommodates a wide range of security workflows -- NDB provisions the infrastructure, the security team handles credential issuance, and NDB completes the integration -- without requiring NDB to hold privileged access to trust infrastructure.

For MongoDB deployments that need to grow beyond their initial topology -- which is often the case eventually -- NDB now provides a clear path that scales the data layer horizontally without sacrificing the robust data protection that makes managed databases worth using.