sdlc.dev

MongoDB path · Scale and change · Topic 10 of 12

Sharding and the shard key

Not started

Sharding splits one collection across several replica sets, called shards, so data and write load can exceed what one machine handles. The application talks to mongos routers, which use metadata from the config servers to send each operation to the shards that own the data. The single most important decision is the shard key: the field or fields whose values decide where each document lives.

Why it matters: The shard key decides which queries hit one shard and which hit all of them, and whether writes spread out or pile onto one machine. Changing it later is possible, but it rewrites the collection.

  1. 1A sharded cluster has three parts

    Shards (each a replica set) hold the data, config servers hold the metadata, and mongos routers send each operation to the right shards.

  2. 2The shard key splits data into ranges

    Documents are grouped into ranges of shard key values (chunks, 128 MB by default), and the balancer moves ranges to even out data size between shards.

  3. 3Good keys have many values, even frequency and no single hot end

    Low cardinality causes jumbo chunks; a value that only grows, such as a timestamp, sends every insert to the last range.

  4. 4Queries without the key go everywhere

    A filter on the shard key (or its prefix) is routed to the shards that own it; anything else is broadcast to all shards and merged.

1

How it works

The explanation, with examples you can run.

The commands below were not run for this path (it needs config servers, several shards and routers); they show the shape of the operations.

Ranged and hashed keys

sh.shardCollection("shop.orders", { customerId: 1, createdAt: 1 })   // ranged, compoundsh.shardCollection("shop.events", { deviceId: "hashed" })            // hashed

With a ranged key, documents with nearby key values live together, so a range query on the key goes to one or a few shards. With a hashed key, MongoDB shards on a hash of the value: writes spread evenly, equality queries are still targeted, but range queries on the key are broadcast to every shard.

Choosing the key

The manual's three questions:

  1. Cardinality. How many distinct values? All documents with one value must sit in one chunk, so few values mean few possible chunks.
  2. Frequency. Is one value far more common than the rest? A very popular tenant or a default value makes one huge chunk.
  3. Monotonic change. Does each new value exceed the last (timestamps, ObjectIds, counters)? Then with ranged sharding every insert goes to the chunk ending at MaxKey, on one shard: a hot shard.

Then check the queries. A query that includes the shard key (or a prefix of it) is routed to the owning shards; one that does not is a scatter-gather across every shard, and the slowest shard sets its latency. A key like { customerId: 1, createdAt: 1 } spreads writes across customers, keeps one customer's orders together, and serves "orders for customer X, newest first" from one shard.

Chunks, the balancer and jumbo chunks

Data is grouped into ranges of key values (chunks), 128 MB by default. The balancer migrates ranges between shards when their data size for a collection differs by more than three times the range size (384 MB at the default). A range that has grown past the limit but contains a single key value cannot be split; this is a jumbo chunk. It stays where it is unless moved by hand, because by default the balancer refuses to migrate a range holding more than about twice the documents a normal range would. The fixes are a better key or a refined one.

Changing your mind

  • Refine (4.4 and later): refineCollectionShardKey adds suffix fields to the existing key, for example { country: 1 } to { country: 1, customerId: 1 }.
  • Reshard (5.0 and later): reshardCollection copies the collection to a new key in the background, then blocks writes briefly (two seconds by default) to switch. It needs spare disk and I/O on every shard.
  • Unshard and move (8.0 and later): unshardCollection turns a sharded collection back into an unsharded one, and moveCollection moves an unsharded collection to another shard.

Every shard is a replica set, so everything in the previous stage (elections, write concern, rollback) applies to each one separately.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Orders are sharded on { createdAt: 1 } with ranged sharding. What happens to inserts?
  2. 2.Sharding on { country: 1 } produces jumbo chunks. Why?
  3. 3.The collection is sharded with a hashed key on customerId. Which query is broadcast to every shard?

0 of 3 right so far. Get all three to finish this topic.

Go deeper