sdlc.dev

Redis path · Keeping it running · Topic 9 of 11

Replication and Sentinel

Not started

A Redis replica is a copy of a primary that follows its changes. It gives you a standby for when the primary's machine fails, and somewhere to send reads. Replication in Redis is asynchronous: the primary replies to the client first and sends the change to replicas afterwards. That is what keeps writes fast, and it is also why "the write was acknowledged" and "the write survived a failover" are different statements.

Why it matters: Replicas keep Redis available when a machine dies, but asynchronous replication means a failover can quietly drop acknowledged writes.

  1. 1Replicas copy the primary

    A replica receives a snapshot, then the stream of write commands, and serves read-only queries.

  2. 2Asynchronous by default

    The primary replies before replicas have the write, so a replica can lag and a failover can lose recent writes.

  3. 3WAIT is not consensus

    WAIT blocks until N replicas acknowledge, which narrows the window but does not make Redis strongly consistent.

  4. 4Sentinel automates failover

    At least three Sentinels watch the primary, agree that it is down, promote a replica and tell clients where the new primary is.

1

How it works

The explanation, with examples you can run.

How a replica follows

# on the replicaREPLICAOF 10.0.0.5 6379

The replica connects, the primary sends a snapshot (an RDB produced by a fork, or streamed straight to the socket with diskless sync), then streams every write command it executes. If the connection drops briefly, the replica asks to continue from its offset in the primary's replication backlog (a partial resync); if it has fallen too far behind, it needs a full resync, another snapshot. Size repl-backlog-size for the outages you expect.

Replicas are read-only by default and can serve reads, but those reads can be slightly stale. INFO replication on the primary shows each replica's offset, and the difference from the primary's offset is how far behind it is.

Narrowing the loss window

Two tools, neither of which turns Redis into a strongly consistent system:

  • WAIT numreplicas timeout blocks the client until that many replicas have acknowledged its earlier writes, or the timeout passes. The replication docs stress that this does not make a set of Redis instances a CP system: a write acknowledged by a replica can still be lost depending on the failover.
  • min-replicas-to-write and min-replicas-max-lag make a primary refuse writes when too few replicas are connected and recently caught up. A primary isolated by a network partition then stops accepting writes after a bounded time, instead of collecting writes that the promoted replica on the other side will never see.

Sentinel

Replication alone does not fail over. Redis Sentinel is a separate process (redis-sentinel, or redis-server --sentinel) that monitors primaries, decides when one is down, promotes a replica, reconfigures the others, and tells clients the new address.

sentinel monitor mymaster 10.0.0.5 6379 2sentinel down-after-milliseconds mymaster 5000sentinel failover-timeout mymaster 60000

The last argument of monitor (here 2) is the quorum: how many Sentinels must agree the primary is unreachable. To actually perform a failover, a Sentinel must also be authorised by a majority of all Sentinels. Hence the advice to run at least three, on machines that fail independently. Clients do not hard-code the primary; they ask Sentinel (SENTINEL get-master-addr-by-name mymaster), and most client libraries support this.

What to expect in a failover

  • A few seconds without a writable primary: detection (down-after-milliseconds) plus election and promotion.
  • Possible loss of writes acknowledged by the old primary but not yet replicated.
  • If the old primary was only partitioned, not dead, it may keep accepting writes from clients on its side until it is reconfigured as a replica, and those writes are discarded. min-replicas-to-write limits this.

So Redis with replicas and Sentinel is highly available, not strongly consistent. For a cache that is fine. For data that must not be lost, keep the source of truth somewhere with synchronous, consensus-based replication.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.A client writes to the primary and gets OK. The primary dies a moment later and Sentinel promotes a replica. Is the write guaranteed to be there?
  2. 2.Why does the Sentinel documentation say to run at least three Sentinels on independent machines?
  3. 3.What does min-replicas-to-write 1 with min-replicas-max-lag 10 do on the primary?

0 of 3 right so far. Get all three to finish this topic.

Go deeper