Troubleshooting TLS Connection Failures Caused by Certificate Expiration

TLS connection failures in Redis Software are often caused by expired or partially updated certificates. These can lead to blocked client traffic, failed UI access, and cluster-wide instability. This article provides complete guidance for managing certificate lifecycles, identifying failure symptoms, renewing certificates, and recovering from full expiration scenarios.

Certificate Lifecycle Management

• Redis uses certificates for TLS/SSL encryption.
• Self-signed certificates typically expire after 12 months.
• Plan and automate rotation for both server-side and client-side certs.
• Use CA-signed certificates in production for OCSP, trust validation, and compatibility.

 Preventing certificate expiration incidents

To prevent unexpected TLS/SSL expirations and avoid downtime, add three layers of protection:

• Visibility: Continuously measure days-until-expiry for every certificate your services present (public and internal), and make the metric easy to query and dashboard.

• Alerting: Notify the right owners well before a certificate expires, with clear thresholds and escalation.

• Automation: Automatically renew and safely reload certificates wherever possible, and verify success with post-renewal checks.

Scope your program to include all relevant certificate types:

• Public-facing HTTPS certificates (web, APIs, CDNs)

• Internal service-to-service certificates (mTLS)

• Load balancer or proxy certificates

• Client certificates, device certificates, and keystores (JKS/PKCS12)

Define ownership for each certificate and ensure contacts are mapped to alert routes and runbooks.

TLS Connection Failure Symptoms

Look for the following errors when certs expire:

• SSLHandshakeException: certificate has expired
• JedisConnectionException, RedisConnectionFailureException
• TLS/SSL handshake or trust chain errors in apps and logs
• Redis CLI, rladmin, or UI becomes unresponsive (if internal certs expire)

Root-Cause Analysis

1. Check expiration of:
   • proxy_cert.pem

   • api_cert.pem

   • syncer_cert.pem
   • cm_cert.pem
   • metrics_exporter_cert.pem

   • ccs_internode_encryption_cert.pem (internal certificate)

   • data_internode_encryption_cert.pem (internal certificate)

   • gossip_ca_signed_cert.pem (internal certificate)

   • mesh_ca_signed_cert.pem (internal certificate)

2. Determine failure scope: mTLS may fail on either client or server expiration.
3. Understand cert type: Self-signed certs often lack revocation and compatibility features.

CA-Managed Internode TLS Certificates

Beginning with Redis Software 8.0.2-17, clusters that use customer-managed internode TLS certificates must ensure all nodes share a consistent CA chain.

If nodes fail to join, replace, or establish intra-cluster TLS handshakes, verify:

• The CA chain is identical across all participating nodes.

• Any rotated or renewed certificates have propagated to every node in the cluster.

Clusters that previously relied on self-signed internode certificates may require updated trust stores when switching to CA-issued certificates.
This verification prevents handshake failures and node rejoin errors after upgrades or certificate rotations.

For configuration and trust-chain details, see Customer-provided certificates for internode encryption

Step-by-Step Certificate Renewal

1. Generate Certificates
   • Self-signed: Use Redis-provided script
   • CA-signed: Generate CSR → submit to CA → receive full chain.
2. Install Certificates
   • Use rladmin cluster certificate set ... for each type
   • Always update all cert types to avoid mismatches.
3. Validate
   • Check Redis UI and CLI for warnings.
   • Ensure all nodes have updated certs.
4. Restart Clients
   • App services or Redis client libraries may need restart to trust the new certs.

Quick Troubleshooting Steps (Emergency)

Use this if the UI and CLI are unresponsive.

1. Detect Expired Certificates

   for file in `find /etc/opt/redislabs -name '*_cert.pem'`; do echo "$file"; openssl x509 -in "$file" -noout -dates; done

   • Monitor proactively using node_cert_expiration_seconds in Prometheus.

2. If the “cm_cert.pem” or “api_cert.pem” are expired

   • This may cause TLS issues when connecting to the UI or making API calls

   • Follow the steps in the “Step-by-Step Certificate Renewal” section

3. Try Internal API Rotation (this must be run on each node)

   curl -l -v -k -u "<user>:<password>" --header 'content-type: application/json' \
   --data '{}' -X POST https://localhost:9443/v1/internal/node/internal_certs/rotate

   • After certificate renewal, restart the services on each node with the command 

     supervisorctl restart all

4. Reinstall & Restart

   rladmin cluster certificate set <cert-name> certificate_file <cert.pem> key_file <key.pem>
   supervisorctl restart all

   • What it does:
     This command updates the specified certificate and restarts Redis services.

   • When to use it:
     Use this only for external certificates such as:

     • cm_cert.pem

     • api_cert.pem

     • proxy_cert.pem

     These must be expired or out of date for the replacement to apply.

     Important limitation:
     This command does not work for internal Redis Enterprise certificates (e.g., for inter-node communication or internal services). Do not use it for those purposes.

     After running the command:

     • Sync the updated certificates across all nodes.

     • Restart services to apply changes.

5. Full Automation (Post-Recovery)

   curl -l -v -k -u "<user>:<password>" -X POST https://<cluster-url>:9443/v1/cluster/certificates/rotate

Troubleshooting Common Issues

Summary: Renewal Commands

Additional Resources

Enabling TLS

Prometheus Monitoring

rladmin Docs