Safe Key Deletion Strategies in Large Redis Clusters

Large Redis OSS Clusters can contain tens of millions of keys. Running FLUSHALL or multi-key DEL in such environments blocks shards, spikes latency, and fragments memory. In production, safer methods are required. This article explains why FLUSHALL and DEL are risky at scale, provides step-by-step strategies for bulk deletion, shows language-specific examples in Python, Bash, and Node.js, highlights a safety checklist for large datasets, and offers troubleshooting guidance for common issues.

Prerequisites

• Redis 4.0 or higher to use UNLINK.

• Access to the cluster via redis-cli or a cluster-aware client (redis-py, ioredis, Jedis, go-redis).

• User credentials with ACL permission to run deletion commands.

• A planned maintenance window or operational safeguards if running in production.

Quick Fix Table

Step-by-Step Strategies

Strategy 1: Use FLUSHALL only for test/dev or downtime windows

FLUSHALL clears all keys on a shard instantly but blocks operations until finished. In OSS Cluster, it must be run on every master shard. Safe only when downtime is acceptable.

Strategy 2: Use SCAN + UNLINK for safe production deletes

• SCAN iterates keys without blocking.

• UNLINK frees memory asynchronously, unlike DEL.

• Run per shard or use a cluster-aware client to avoid CROSSSLOT errors.

Strategy 3: Use pipelined deletion for very large datasets

For tens of millions of keys, pipeline UNLINK calls in batches to reduce round-trip latency and complete deletes faster.

Examples

Python (redis-py, RedisCluster)

from redis.cluster import RedisCluster
startup_nodes = [{"host": "127.0.0.1", "port": 7000}]
rc = RedisCluster(startup_nodes=startup_nodes, decode_responses=True)
def delete_keys_by_pattern(pattern="user:*", scan_count=1000, pipe_batch=1000):
   cursor = "0"
   while True:
       cursor, keys = rc.scan(cursor=cursor, match=pattern, count=scan_count)
       if keys:
           for i in range(0, len(keys), pipe_batch):
               batch = keys[i:i+pipe_batch]
               pipe = rc.pipeline()
               for k in batch:
                   pipe.unlink(k)
               pipe.execute()
       if cursor == "0":
           break
delete_keys_by_pattern(pattern="user:*")

Bash (redis-cli, per-shard SCAN + UNLINK)

#!/usr/bin/env bash
set -euo pipefail

CLUSTER_HOST=127.0.0.1
CLUSTER_PORT=7000
PATTERN="user:*"
BATCH_SIZE=1000

mapfile -t MASTERS < <(
  redis-cli -c -h "$CLUSTER_HOST" -p "$CLUSTER_PORT" CLUSTER SLOTS \
  | awk '/"/{gsub(/"/,"")} {for(i=1;i<=NF;i++){if($i~/^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/){ip=$i;port=$(i+1);if(port~/^[0-9]+$/){print ip":"port}}}}' \
  | sort -u
)

for master in "${MASTERS[@]}"; do
  IFS=':' read -r HOST PORT <<< "$master"
  echo "Processing shard $HOST:$PORT"
  CURSOR=0
  while : ; do
    read -r CURSOR KEYS <<< "$(
      redis-cli -h "$HOST" -p "$PORT" --raw SCAN "$CURSOR" MATCH "$PATTERN" COUNT "$BATCH_SIZE" \
      | awk 'NR==1{c=$0;next}{k=k?k" "$0:$0}END{print c, k}'
    )"
    if [[ -n "$KEYS" ]]; then
      printf '%s\n' $KEYS | xargs -r -n "$BATCH_SIZE" redis-cli -h "$HOST" -p "$PORT" UNLINK
    fi
    [[ "$CURSOR" == "0" ]] && break
  done
done

Node.js (ioredis Cluster)

const IORedis = require("ioredis");

const cluster = new IORedis.Cluster([{ host: "127.0.0.1", port: 7000 }]);
const PATTERN = "user:*";
const BATCH_SIZE = 1000;

async function unlinkBatch(node, keys) {
  if (!keys.length) return;
  const pipe = node.pipeline();
  for (const k of keys) pipe.unlink(k);
  await pipe.exec();
}

async function scanAndDelete(node) {
  let cursor = "0";
  do {
    const [next, keys] = await node.scan(cursor, "MATCH", PATTERN, "COUNT", String(BATCH_SIZE));
    if (keys && keys.length) {
      for (let i = 0; i < keys.length; i += BATCH_SIZE) {
        await unlinkBatch(node, keys.slice(i, i + BATCH_SIZE));
      }
    }
    cursor = next;
  } while (cursor !== "0");
}

(async () => {
  const masters = cluster.nodes("master");
  for (const node of masters) {
    console.log(`Processing shard ${node.options.host}:${node.options.port}`);
    await scanAndDelete(node);
  }
  await cluster.quit();
})();

Tuning and Safety Checklist

• Batch sizing: Start small (500–1000 keys per batch) and adjust upward.

• Concurrency: Run sequentially per shard first; only parallelize if cluster resources allow.

• Fragmentation: After huge deletes, run MEMORY PURGE or schedule a controlled restart.

• Persistence impact: Expect large AOF/RDB rewrites after deletion; monitor disk and replication.

• Monitoring: Track latency, ops/sec, CPU, and memory usage during deletion.

Troubleshooting

Recommended patterns at a glance