Troubleshooting Active-Active (CRDB) Sync Failures After Upgrade or DR Events

Redis Software Active-Active databases (CRDBs) use the syncer process and inter-cluster replication to maintain data consistency across participating clusters. Each cluster hosts its own database while only the data is replicated, so configuration, module versions, and certificates must remain aligned across all participants.

After events such as Redis Software upgrades, disaster recovery drills, cluster reboots, certificate rotations, or participant re-creation, CRDB synchronization can stop progressing even when databases appear healthy. This guide helps you quickly identify the root cause and apply the correct fix using scenario-based navigation, Quick Fix actions, and targeted recovery steps.

Start Here: Identify Your Scenario

Use this to jump directly to the most relevant fix:

• After a Redis Software upgrade → Go to Verify version alignment

• After a cluster reboot or restart → Go to Restart sync safely

• After DR rebuild or participant re-creation → Go to Fix DR configuration drift

• After certificate or mTLS change → Go to Resolve certificate issues

• Links disconnected or health-report hangs → Go to Validate connectivity

Quick Fix

Prerequisites

• Administrative access to all participating clusters

• Access to crdb-cli, rladmin, openssl, and REST API

• CRDB GUID and database IDs

• Maintenance window for disruptive actions

Important:

• Inter-cluster connectivity must be functional before sync can recover

• Full sync operations can generate significant traffic

• Certificate updates must be consistent across all clusters

• Avoid repeated forced sync attempts before identifying root cause

Fix Connectivity Issues Between CRDB Participants

When to use:

• health-report hangs

• Links show disconnected

• One participant appears unreachable

Run:

openssl s_client -connect <endpoint>:<port> -servername <endpoint>

Optional:

openssl s_client -connect <endpoint>:<port> -servername <endpoint> -showcerts

Interpret results:

• Connection failure → check firewall, routing, DNS

• No listener → validate endpoint and port

• Certificate mismatch → TLS trust issue or, particularly for Kubernetes or if a proxy is in use, make sure that your route/ingress is configured for TLS passthrough. Replication will not work if your route/ingress is presenting a different certificate than the cluster.

If using a load balancer:

• Confirm correct backend mapping

• Confirm that TLS passthrough is enabled

• Ensure CRDB ports are exposed

Verify Version Alignment Across Clusters

When to use:

• Issue started after upgrade

Run:

rladmin status extra all

Expected: All clusters run the same Redis Software version

If not:

• Complete upgrade across all clusters

• Do not attempt CRDB fixes until aligned

Then run:

crdb-cli crdb update --crdb-guid <CRDB_GUID> --update-db-config-modules true

This command does not finish the module step on its own. It updates module information in the global Active-Active configuration (default_db_config) only. Each participating instance also has its own db_config, which overrides default_db_config, and this command does not touch it. The command still exits successfully and reports Status changed: started -> finished either way, so there is no signal that anything was missed.

This applies only to Active-Active databases whose module configuration sets module_id or semantic_version. Those fields became optional in v7.4.2 and are deprecated as of v7.8.2, but databases created earlier still carry them, and the module version recorded at creation is never updated. To check, retrieve the configuration and review every instance, not just the default:

curl -k -u "<user>:<password>" https://<cluster-fqdn>:9443/v1/crdbs/<crdb-guid>

If no module_list entry contains module_id or semantic_version, each cluster resolves its own module versions and there is nothing further to do. If entries do contain them, compare the versions in default_db_config.module_list and in every instances[].db_config.module_list against the versions the databases actually run, from GET /v1/bdbs/<bdb-uid>. Instances left on an older module version can cause a later attempt to add or remove a participating cluster to fail. Contact Redis Support if they do not match.

Confirm Cluster and Database Health

Run:

rladmin status extra all
rladmin cluster running_actions

Fix before continuing if:

• Shards are down or recovering

• The database reports an active or stuck change in progress

• Actions are stuck

CRDB sync requires healthy local databases.

Resolve Certificate or mTLS Issues

When to use:

• Sync failed after certificate rotation

• mTLS enabled or changed

• Syncer logs show certificate errors

Steps:

• Update certificates using supported methods (UI, rladmin, or API)

• Ensure all clusters trust the same CA

• Verify SAN matches endpoint

Then use the following command to sync the new certificates into the CRDB config:

crdb-cli crdb update --crdb-guid <CRDB_GUID> --force

Restart Sync Safely

When to use:

• Sync stopped after reboot or failure

Option A:

curl -k -u <user>:<pass> -X PUT \
  -H "Content-Type: application/json" \
  -d '{"sync":"enabled"}' \
  https://<host>:<port>/v1/bdbs/<database-id>

Option B:

crdb-cli crdb update --crdb-guid <CRDB_GUID> --force

Only perform after resolving the root cause.

Fix DR Configuration Drift

When to use:

• Participant was deleted and recreated

• Database version or modules do not match

Check the database configuration and CRDB configuration to see if they match (Redis compatibility version, number of shards, size, modules, etc.):

crdb-cli crdb get --crdb-guid <CRDB-GUID>

Update CRDB metadata:

crdb-cli crdb update --crdb-guid <CRDB_GUID> \
  --default-db-config '{"version":"7.4"}'

If modules differ, do not write the module list by hand. Let the cluster read the module versions the database actually runs:

crdb-cli crdb update --crdb-guid <CRDB_GUID> --update-db-config-modules true

Then check the per-instance configuration as described in Verify version alignment across clusters, because that command updates default_db_config only.

Then align the database locally if needed.

Evaluate Replication Backlog Impact

If sync was down for an extended period:

• Expect higher bandwidth and longer recovery

Avoid repeated retries while unstable.

Validate Recovery

Run:

crdb-cli crdb health-report --crdb-guid <CRDB_GUID>
rladmin status extra all

Expected result:

• All links connected

• No shard errors

• Sync lag decreasing

• No TLS or connectivity errors in logs

Common Mistakes to Avoid

• Repeatedly running --force without fixing root cause

• Upgrading only some clusters

• Applying certificate changes without running a crdb update

• Recreating DR participants without validating configuration

• Hand-writing module_list with an explicit semantic_version, or with module_args: "PARTITIONS AUTO". Both are deprecated, and either will cause the configuration to break again on a later upgrade or participant change.

What to Expect During Recovery

• Partial sync → fast recovery

• Full sync → longer recovery, higher bandwidth

• Cross-region sync → slower convergence

When to Contact Support

If the issue persists, collect:

• crdb-cli crdb health-report output

• rladmin status extra all

• rladmin cluster running_actions

• Syncer and proxy logs

• Support packages from all clusters

• Timeline of triggering event

Notes

• Active-Active requires both connectivity and configuration consistency

• Certificate and metadata drift are the most common causes after upgrade and DR events

• Use this article as the primary recovery flow before escalating