Recover an UNREACHABLE node in a Redis cluster

In an existing Redis Software cluster, a node marked as UNREACHABLE indicates that the cluster cannot communicate with that node or retrieve shard state. This can impact availability, block configuration changes, and degrade cluster health. This article explains how to diagnose node-level failures and safely recover using Step-by-Step Instructions, along with guidance for Troubleshooting and node replacement.

Note: this procedure is for one UNREACHABLE node only; if multiple nodes are UNREACHABLE or the cluster loses majority, they must stop and contact Redis Support.

Quick Fix

Prerequisites

• Cluster admin access

• SSH access to nodes

• Tools: rladmin, rlcheck, supervisorctl

Step-by-Step Instructions

1. Identify affected node

rladmin status
rladmin status shards

• Identify node not OK

• Identify shards hosted on that node

2. Check node OS health

df -h
free -m
timedatectl

• Disk < 90%

• No memory pressure

• Time sync correct

3. Check Redis services

supervisorctl status
rlcheck

• All services must be RUNNING

• Fix any reported errors

4. Verify connectivity

From another node:

ping <node-ip>
curl -k https://<node-ip>:9443

5. Revalidate cluster

rladmin status
rladmin status shards

6. Replace/Remove failed node if needed

If unrecoverable:

• Provision new node (same version)

• Join cluster

• Move shards

• Remove failed node

Troubleshooting

Key takeaways

• UNREACHABLE = communication or service failure

• Always fix node before retrying operations

• Replace node if recovery is not fast or safe