sdlc.dev

MongoDB path · Writes you can trust · Topic 7 of 12

Replica sets, elections and failover

Not started

A replica set is a group of mongod processes holding the same data. One member, the primary, accepts writes and records each change in its oplog, a capped collection of operations. The secondaries pull the oplog and apply it, a little behind. If the primary disappears, the remaining members elect a new one. Production deployments use replica sets even on a single shard, and features such as transactions and change streams require one.

Why it matters: Failover is automatic, but it is not instant and not free: writes stop for seconds, and writes the old primary never replicated can be rolled back. Applications must be ready for both.

  1. 1One primary takes writes

    Secondaries copy the primary's oplog, an ordered log of changes, and apply it to their own data.

  2. 2A majority elects the next primary

    If the primary is unreachable for electionTimeoutMillis (10 s by default), a secondary that can reach a majority of voters calls an election.

  3. 3Writes pause during failover

    Until a new primary is elected, writes fail. Drivers retry a write once (retryable writes) and rediscover the new primary.

  4. 4Unreplicated writes roll back

    A write the old primary accepted but no majority received is undone when it rejoins, unless it was written with w majority.

1

How it works

The explanation, with examples you can run.

What the set looks like

rs.initiate({ _id: "rs0", members: [  { _id: 0, host: "mongo1:27017" }, { _id: 1, host: "mongo2:27017" }, { _id: 2, host: "mongo3:27017" }] })rs.status().members.map(m => m.name + " " + m.stateStr)
[ 'mongo1:27017 PRIMARY', 'mongo2:27017 SECONDARY', 'mongo3:27017 SECONDARY' ]

A set can have up to 50 members, of which at most 7 vote. Use an odd number of voters so that a majority exists on one side of any split.

Elections

Members send each other heartbeats every two seconds. When a secondary has not heard from the primary for electionTimeoutMillis (10,000 ms by default), it raises the term and asks for votes. The first candidate to win a majority becomes primary; members prefer candidates with the most recent oplog and higher priority. MongoDB's documentation puts the median time to elect a new primary at no more than about 12 seconds with default settings.

Two failovers on the test set show the difference between a clean shutdown and a crash:

docker stop  (clean shutdown)  14:05:17 stop sent  -> 14:05:18 new primary, term 2docker kill  (crash)           14:05:35 kill sent  -> 14:05:44 new primary, term 3

A primary shutting down cleanly steps down and lets a caught-up secondary take over at once. A crashed one has to be noticed first, which costs most of the election timeout. A primary that can see only a minority of voters steps down by itself, so a partitioned old primary stops accepting writes.

What applications see

During those seconds, writes fail with errors such as NotWritablePrimary and reads that require the primary fail too. Drivers since the 4.2 generation enable retryable writes: they retry an eligible write once after a network error or failover, and the server uses a transaction number to avoid applying it twice. Set your timeouts so a 10 to 15 second gap degrades gracefully instead of cascading.

Rollback

If the old primary accepted writes that no other member received, the new primary's history does not include them. When the old member rejoins, it rolls those writes back to match, saving them to rollback files for manual inspection. The client was told they succeeded. Writing with w: "majority" (the next topic) means an acknowledged write is on a majority and survives any election.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.A three-member replica set is split by the network: the primary on one side, both secondaries on the other. What happens?
  2. 2.Why might you prefer three data-bearing members to two members plus an arbiter?
  3. 3.A write succeeded with w: 1. Two seconds later the primary crashed before any secondary copied it. What happens to the write?

0 of 3 right so far. Get all three to finish this topic.

Go deeper