MySQL on NDB Architecture Deep Dive: From HA Foundation to Recovery and Security Operations

The Nutanix Database Service (NDB) solution offers a highly automated MySQL High-Availability (HA) topology backed by group-replication consensus, a redundant router tier, and NDB Time Machine for point-in-time protection. This blog walks through the architecture of MySQL on NDB and the end-to-end lifecycle of the core operations exposed to platform users: provisioning, backup, restore, clone, point-in-time recovery, Transparent Data Encryption (TDE) enablement, and TDE master-key rotation.

MySQL HA Architecture on NDB

NDB deploys a MySQL HA database as a group of three or more database VMs (DBServers) configured in a single group-replicated cluster. One node is elected primary and accepts writes; the remaining nodes replicate synchronously via a Paxos-style¹ protocol and serve as read replicas.

Building blocks

• NDB Control Plane — the control-plane service that is designed to orchestrate operations  with structured execution steps, progress reporting, and recovery hooks.
• NDB DBServer — a VM that runs the MySQL engine plus a lightweight NDB agent. Storage is provisioned on Nutanix Volume Groups for data (datadir) and binary logs (binlogdir).
• Group-Replicated Cluster — the set of DBServers joined into a single replication group; NDB manages the cluster-admin account used to add and remove members.
• NDB Router Nodes — (optional) up to 2 lightweight proxy VMs that terminate application traffic and redirect it to the currently-elected primary for writes or an online secondary for reads.
• NDB Time Machine — the per-database container that orchestrates snapshots, binlog archival to an Object Store, and point-in-time replay for restore and clone operations.

 ¹Paxos is a family of protocols for solving consensus in a network of unreliable or fallible processors - see here for more information.

Roles and Failure Behaviors

Connection Topology & Optional MySQL Router

Applications should not be pinned to a specific MySQL node - especially in an HA topology where the primary can shift after a failover. NDB supports two topologies:

Direct connection (Single Instance)

For Single Instance (SI) databases, clients connect directly to the DBServer's IP on port 3306. NDB registers the instance with the NDB control plane, records credentials in the entity credential store, and exposes the endpoint via the database detail API. No router layer is deployed.

Router-fronted connection (HA / Group Replication)

For HA databases, NDB can bootstrap MySQL Router against the Group Replication (GR) cluster. The NDB solution is designed to avoid talking to the cluster through a floating VIP; instead it fronts the cluster with a small pool of router nodes that transparently direct traffic based on the current group topology.

Router sits between the client and the cluster and transparently splits traffic across four ports:

• RW port (default 6446) - classic² protocol, primary member.
• RO port (default 6447) - classic protocol, load-balanced across secondaries.
• X-RW port (default 6448) - X protocol, primary member.
• X-RO port (default 6449) - X protocol, secondaries.

²See MySQL Protocols for in-depth explanation of various protocols

Router responsibilities

• Bootstrap against the group-replicated cluster using an NDB managed cluster-admin account.
• Publish four listener ports: classic Read-Write, classic Read-Only, X-protocol Read-Write, and X-protocol Read-Only.
• Continuously refresh cluster topology so newly-added secondaries automatically join the RO pool and offline nodes are drained.
• Persist its bootstrapped credentials in an on-disk keyring with secured ownership so that only the NDB managed router service account can read them.
• Restart cleanly under systemd, and be re-bootstrapped idempotently by NDB when the cluster admin credentials rotate.

The router is optional. Users can opt out of the router and connect directly to the Group Replication cluster if they prefer to use alternatives, such as (HAProxy, ProxySQL, cloud LBs) - NDB does not require it as long as the cluster itself is healthy.

Provisioning a MySQL HA Instance on NDB

Provisioning is the workflow that turns a set of virtual machines and profiles into a fully-configured, group-replicated MySQL cluster registered with NDB. NDB’s control plane is designed to orchestrate key steps of this process: profile lookup, VM creation, storage attachment, cluster bootstrap, router deployment, and database registration.

Inputs consumed by NDB

NDB will use the following inputs to enable the management and execution of the workflows it provides:

• Software, compute, network, and DB-parameter profiles³.
• Number of DBServer nodes (single or clustered).
• Optional pool of router VMs to deploy alongside the cluster.
• SLA selection — if PITR-enabled, NDB requires an Object Store⁴ to be registered for binlog archival.
• TDE flags (enable TDE, KMS ID, KMS config ID) and log encryption preferences for binlog, redo, and undo.

End-to-end workflow

The provisioning process involves a series of automated, orchestrated steps executed by the NDB control plane designed to help establish a fully-configured, high-availability cluster:

• Pre-processing: NDB validates profile compatibility, checks TDE prerequisites, and confirms that PITR SLAs have a registered Object Store when the cluster is HA.
• Create Virtual Machines: NDB provisions the DBServer VMs (and router VMs, if requested) with the selected compute and network profiles.
• Network reachability: NDB verifies that the target VMs are reachable and updates step names to include the freshly allocated IPs for operator visibility.
• Restart Network Services: NDB normalizes network configuration and ensures each host's /etc/hosts contains every other DBServer hostname.
• Attach storage: NDB creates and attaches dedicated volume groups for the data directory and the binary log directory according to the storage layout.
• Apply best practices: VM-level and MySQL-level best practices are applied from an NDB managed template.
• Configure MySQL: NDB installs and configures the MySQL server binaries, prepares TDE-related plugins and log-encryption flags if requested.
• Bootstrap the group: NDB creates the group-replication cluster on the elected primary, then adds each secondary — with retry-on-failure and rescans of the cluster metadata.
• Deploy the router tier (optional): if router nodes were requested, NDB configures and bootstraps each router against the freshly created cluster.
• Register the database: NDB records the database, the DBServer logical cluster, and — if TDE is enabled — seeds the keyring metadata used by later key-rotation operations.
• Collect logs and rollback on failure: on any failure, NDB stops the instance, zips diagnostic logs, and drives a rollback that detaches storage and cleans up the partially-created cluster.

³See Nutanix Database Service Profiles for in-depth explanations of these profiles.
⁴See Nutanix Object Stores for in-depth explanation of the objects.

Backup: Snapshot and Log Catchup with Time Machine

NDB Time Machine captures two flavors of backup for a MySQL HA database: snapshots (storage-consistent point captures of the DBServer volumes) and log catchup archives (binlogs and metadata continuously uploaded between snapshots). Together they deliver both scheduled protection and on-demand PITR.

Snapshot workflow

• Preprocessing: NDB validates the workflow inputs, runs backup-operation guardrails on the MySQL node, and revalidates the Primary node against the live group-replication cluster. If leadership has moved, the operation is redispatched to the correct node.
• Application metadata refresh: NDB refreshes its cached view of the database (parameters, node topology, keyring metadata).
• Software snapshot: a snapshot of the MySQL software volume is created (or skipped and re-linked if the previous one is still current).
• Quiesce: a lightweight, best-effort quiesce is issued to the primary — non-blocking, since InnoDB is crash-consistent and the storage snapshot is atomic.
• Database snapshot: the data and binlog volume groups are snapshotted through the Nutanix data path.
• Unquiesce: a no-op for InnoDB because quiesce has an implicit timeout.
• Upload snapshot / GTID capture: for special or first-of-schedule snapshots, NDB persists the executed GTID set as a database property so log-catchup can resume replay from the exact same position.

Why not mysqldump?

Storage-level snapshots combined with binary logs are designed to perform significantly faster than traditional logical utilities for anything non-trivial, and restores are designed to be highly efficient because they reuse the source disk chain. Logical backups are still useful for cross-version migrations and portability, but they are not the primary protection primitive NDB relies on.

Log catchup

Between snapshots, NDB continuously drains new binary log events from the primary DBServer and uploads them, in epoch-sized batches, to the NDB Time Machine's Object Store target. The size of each epoch is a tunable NDB setting — larger epochs reduce upload overhead, smaller epochs reduce the worst-case PITR replay time.

• Log catchup is guarded by the same Primary node revalidation and backup guardrails as the snapshot workflow.
• Failed uploads are retried; successful uploads advance the captured GTID watermark stored on the database.
• PITR-enabled SLAs require a registered Object Store — NDB refuses to provision a MySQL HA database with a PITR SLA when no Object Store is available.

Consistency contract

Each snapshot is stamped with the executed GTID set at the moment of capture. Log catchup archives every subsequent GTID in strict order. Any restore or clone can therefore be replayed deterministically from any snapshot up to any archived GTID — translating directly into a wall-clock PITR timestamp.

Restore: Snapshot and Point-in-Time Recovery

Restore rehydrates an existing MySQL HA database to a chosen snapshot or PITR timestamp in place. NDB owns the orchestration across every node in the cluster, taking a safety snapshot first, replacing volumes, and reconciling group membership at the end.

Restore is supported for HA only

Single-instance MySQL databases on NDB do not use this workflow — they follow the simpler in-place recovery path. PITR restore of a MySQL HA database additionally requires the NDB Time Machine to have the modern log-drive-container layout enabled; older NDB Time Machines must be migrated before PITR restore becomes available.

Top-level workflow

• Preprocessing: fetch and validate the physical and logical DBServer clusters; probe everynode's reachability and confirm a quorum is available.
• Elect primary: designate the primary DBServer that will be restored first and act as the source for resynchronizing the secondaries.
• Replicate snapshot: if the chosen snapshot lives on a different Nutanix cluster than the target DBServers, NDB replicates it to the local cluster.
• Delete pre-restore snapshots: clean out any stale pre-restore safety snapshots from previous runs.
• Stop all nodes: gracefully stop MySQL on every DBServer to avoid split-brain during storage replacement.
• Restore primary: run the following primary sub-operations on Primary:
  • Take a pre-restore safety snapshot so the operation is reversible.
  • Detach the existing data storage.
  • Restore the chosen snapshot's volume group in its place.
  • Recover MySQL — for PITR, replay archived binlogs up to the target GTID/timestamp; for snapshot restore, simply start the engine.
  • Refresh application metadata inside NDB.
  • Take a fresh snapshot of the newly-restored primary so secondaries can rebuild from a consistent Baseline.
  • Post-processing hooks (parameter reapply, TDE reconcile, etc.).
• Restore secondaries: run the following secondary sub-operations on each remaining DBServer in parallel:
  • Detach existing storage on the secondary.
  • Restore from the primary's freshly-taken post-restore snapshot.
  • Recover MySQL and rejoin the group as an ONLINE member.
  • Refresh metadata; run post-processing hooks.
• Post-processing: re-establish group replication, refresh cached metadata, and re-elect the primary role inside NDB.

Rollback

Because the workflow is designed to capture a pre-restore safety snapshot before detaching storage, a failed restore can be rolled back by reversing the detach/restore steps against that safety snapshot — no data loss for the pre-restore state.

Encrypted binlog replay

If the source has binlog encryption enabled, the tde_enabled flag is threaded into the recovery input so the driver knows to preserve the keyring context needed to decrypt those binary logs during replay. Skipping that would silently produce a clone that stops at the last plaintext log file, a subtle failure mode worth calling out.

Clone from NDB Time Machine

Note: Cloning is currently supported for SI to SI clones. Support for HA clones is expected in an upcoming release.

Cloning creates a brand-new MySQL database from an NDB Time Machine — either from a specific snapshot or from a chosen PITR timestamp. The source database is untouched. NDB provisions a fresh DBServer VM or clones into DBServer being associated with the time machine, restores the chosen state onto them, and registers the clone as an independent database.

Workflow at a glance

• Preprocessing: Load the source NDB Time Machine metadata, credentials, and connection parameters.
• Provision new or into associated target DB server: Allocate the new DBServer VM using the requested profile in case of new provisioning or use the existing DBServer VM being associated with Time Machine.
• Replicate the source snapshot if it lives on a different Nutanix cluster than the target VMs.
• Restore snapshot + PITR replay: Apply the source snapshot to the new database server VM, replay binlogs up to the requested timestamp (if any).
• Register the clone as a new database with its own credentials.

Snapshot vs PITR clones

A snapshot clone is designed to offer a highly efficient path — it uses only the chosen snapshot. A PITR clone is snapshot + binlog replay, so it scales with the distance between the base snapshot and the requested timestamp; smaller log-catchup epochs reduce replay time.

Transparent Data Encryption (TDE) for MySQL

NDB supports enabling MySQL Transparent Data Encryption at provisioning time. TDE encrypts InnoDB tablespaces, and — optionally — the binary logs, InnoDB redo log, and InnoDB undo log. Keys are supplied by an external, KMIP-compatible Key Management Server (KMS) that NDB has been previously configured with.

Prerequisites enforced by NDB

• TDE is only available on software profiles created with a TDE-capable MySQL Enterprise binary that includes the external-KMS keyring plugin. Older profiles are rejected at provisioning time.
• Both a KMS ID and a KMS configuration ID must be supplied when TDE is requested.
• TDE-with-external-KMS support must be enabled via NDB's management-plane feature flags; if disabled, the provisioning workflow fails fast with a clear message.
• Log-encryption preferences default to ON for binlog, redo, and undo when TDE is enabled — each can be explicitly set to OFF via the provisioning input.

KMS material placement

For each TDE-enabled DBServer, NDB writes the KMS client configuration and TLS material into a dedicated, engine-specific configuration directory. Three certificate files are required for mutual TLS to the KMS: a CA certificate, a client certificate, and a client private key (optionally password-protected). NDB manages ownership and permissions of that directory so only the MySQL OS user can read it; designed to help protect against secrets being logged.

Log-layer encryption switches

In addition to InnoDB tablespace encryption, the provisioning flow honors three log-encryption flags. Defaults are ON when TDE is enabled; each can be individually turned OFF via the request:

• enable_binlog_encryption - encrypts binary and relay logs. Designed to help support secure PITR protection when data is encrypted at rest.
• enable_innodb_redo_log_encrypt - encrypts the InnoDB redo log.
• enable_innodb_undo_log_encrypt - encrypts the InnoDB undo log.

What TDE protects

• InnoDB tablespace key — encrypts on-disk table data.
• Replication key — encrypts binary logs and relay logs so replicated transactions are also encrypted at rest.
• InnoDB redo/undo log encryption — optional, enabled by default when TDE is on.

All keys are stored inside the MySQL keyring and wrapped by a master key that is fetched from the external KMS on server start.

Master key rotation on NDB

NDB exposes a first-class "rotate master key" operation for TDE-enabled databases. It rotates the InnoDB master key (which re-encrypts tablespace keys) and/or the replication key (which re-encrypts binlogs and relay logs), coordinates the rotation across the whole HA cluster, and keeps NDB Time Machine in a safe state.

Master Key rotation flow

• Pre-processing: fetch the current node topology from NDB; validate every cluster member reports ONLINE for group replication; resolve the live primary by asking each node for its own group-replication role (hostnames may not match DBServer names after normalization).
• Pause NDB Time Machine: temporarily block scheduled backups so the rotation completes atomically from the NDB Time Machine's perspective.
• InnoDB key rotation (if requested): rotate the InnoDB master key on the primary, then update the cached InnoDB key metadata for every node from the local keyring.
• Replication key rotation (if requested): rotate the replication key on the primary first; then rotate on each secondary — a secondary failure is logged as a warning and the workflow continues (the primary's key is authoritative).
• Invalidate previous backups: mark existing snapshots as unusable for restore, because their encrypted contents no longer align with the new master key.
• Resume NDB Time Machine: re-enable scheduling and trigger a fresh snapshot so a new, valid protection baseline exists before the operation returns success.
• Rollback on failure: if any step fails, NDB drives a rollback that unwinds the rotation state and records the failure — NDB Time Machine operations are designed to resume so the database is left protected.

Single-instance vs HA

Why invalidate backups?

The master key wraps the keys that were used to encrypt data on disk. After rotation of the replication key, binlog backups taken with the old master key can no longer be restored on a database because MySQL deletes the old replication key from the KMS. NDB therefore invalidates them and immediately captures a fresh snapshot so protection is restored before the operation completes.

Rotation is not reversible

MySQL's ALTER INSTANCE ROTATE ... MASTER KEY is one-way. The orchestrator's rollback handler therefore does not attempt to "un-rotate" - it invalidates the now-orphaned backups and resumes the NDB Time Machine so protection continues with the new key generation. That failure mode should be understood before scheduling rotations against critical HA clusters.

Operational Guardrails and Feature Toggles

Every MySQL HA workflow on NDB is gated by explicit management-plane toggles so that operators can safely stage the rollout of new capability, or disable a subsystem in response to a field incident without a redeploy.

Conclusion

MySQL on NDB is more than a wrapper around 'apt install mysql-server' or "MySQL plus a UI”. It is a set of opinionated, workflow-driven orchestrations that layer HA topology, NDB Time Machine, and TDE into a single managed experience — with the guardrails, rollbacks, and feature toggles required to run it safely in production.

The value shows up in the seams: quiesce-aware snapshots that survive crashes, PITR that understands GTIDs and encrypted binlogs, an HA cluster that treats each member as a discoverable, precheck-gated InstanceDefn, and a TDE story where the platform owns KMS trust while MySQL owns the keys.

©2026 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and all Nutanix product and service names mentioned are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. All other brand names mentioned are for identification purposes only and may be the trademarks of their respective holder(s).

This content may contain express and implied forward-looking statements, including but not limited to statements regarding our plans and expectations relating to new product features, including HA-cloning support, and technology under development, the capabilities of such product features and technology, and our plans to release product features and technology. Such statements are not historical facts and are instead based on our current expectations, estimates and beliefs, including statements about The accuracy of such statements involves risks and uncertainties and depends upon future events, including those that may be beyond our control, and actual results may differ materially and adversely from those anticipated or implied by such statements, including, among others: failure to develop, or unexpected difficulties, delays or disruptions in developing, releasing or distributing, new products, services, product features or technology in a timely or cost-effective basis. Any forward-looking statements included speak only as of the date hereof and, except as required by law, we assume no obligation to update or otherwise revise any such forward-looking statements to reflect subsequent events or circumstances. Certain products and features or functionalities described herein remain in varying stages of development and will be offered on a when-and-if-available basis. The development, release, and timing of any such products, features or functionalities are subject to change. Nutanix will not have any liability for any failure to deliver or delay in the delivery of any such products, features or functionalities. Any future product or product feature information is intended to outline general product directions, and is not a commitment, promise, or legal obligation for Nutanix to deliver any functionality. This information should not be used when making a purchasing decision.