sdlc.dev

Redis path · Keeping it running · Topic 11 of 11

Memory sizing

Not started

Everything Redis stores is in memory, so the size of the machine is the size of the database. Sizing is more than adding up value sizes: each key has overhead, client and replication buffers take space, background saves need extra memory while they run, and the allocator rarely hands freed memory back neatly. Most Redis incidents involving memory come from a plan that only counted the data.

Why it matters: Redis's dataset lives in RAM; running out means evictions, refused writes or the process being killed, usually at peak traffic.

  1. 1Data is not the only cost

    Each key carries overhead, and buffers for clients, replication and the AOF also use memory.

  2. 2Measure, then multiply

    Load a realistic sample, read used_memory and MEMORY USAGE, and scale up; do not estimate from the raw size of your values.

  3. 3Leave headroom

    Set maxmemory well below the machine's RAM, to leave room for copy-on-write during saves, buffers and fragmentation.

  4. 4Fragmentation is real

    RSS can stay higher than used memory after many deletes; mem_fragmentation_ratio shows how much.

1

How it works

The explanation, with examples you can run.

Where memory goes

  • Keys and values, plus per-key overhead: the key string, the entry in the main dictionary, object headers, expiry metadata, and allocator rounding. For small values the overhead can be larger than the value.
  • Encodings: small hashes, lists, sets and sorted sets use compact listpacks; past their thresholds they convert to larger structures. Grouping many tiny values as fields of hashes can save a lot of memory.
  • Buffers: each client's output buffer (a slow pub/sub subscriber or a huge reply can grow it), the replication backlog, and buffers for replicas being synchronised.
  • Copy-on-write during forks: a snapshot or AOF rewrite shares memory with the parent, but every page written during the save is duplicated.
  • Fragmentation: after many deletes and resizes, the process holds more memory (RSS) than Redis is using for data.

Measure it

Load a realistic sample, then look:

> MEMORY USAGE user:42(integer) 84> MEMORY USAGE user:42:name(integer) 48

Even a three-letter string costs 48 bytes once its key and overhead are counted. For the whole instance:

> INFO memoryused_memory_human:2.00Mmaxmemory_policy:noevictionmem_fragmentation_ratio:1.74...

(These figures come from a small test instance capped at 2 MB; on datasets that small, fixed overhead dominates and the fragmentation ratio means little.) Multiply the sample's per-key figure by the expected key count, add the buffers and growth you expect, then add headroom. redis-cli --bigkeys and --memkeys scan for the largest keys without blocking the server for long.

Set limits

maxmemory 10gbmaxmemory-policy allkeys-lru     # for a cache; noeviction for data you must keep

maxmemory limits the dataset, not everything the process uses, so set it below the machine's RAM: buffers, fragmentation and copy-on-write all sit on top. How much below depends on the write rate during saves; leaving a large share free on machines that fork for persistence is common practice. With noeviction, reaching the limit means writes fail with an OOM error, which is right for data that must not be silently dropped and wrong for a cache.

Keep it in check

  • Give cache keys TTLs, and bound collections that grow (LTRIM, XADD ... MAXLEN, ZREMRANGEBYRANK).
  • Avoid very large single keys; they are slow to delete (use UNLINK), to migrate between cluster nodes and to replicate.
  • Watch used_memory, evicted_keys, mem_fragmentation_ratio and client output buffers over time, and alert well before the limit.
  • When one machine is not enough, shard with Redis Cluster rather than buying ever larger hosts; smaller primaries also fork and restart faster.
2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.You store 10 million keys whose values are 20 bytes each. Is about 200 MB of RAM enough?
  2. 2.A 16 GB machine runs Redis with persistence on. Why not set maxmemory to 16gb?
  3. 3.INFO memory shows used_memory 2 GB and mem_fragmentation_ratio 1.8 after a large batch of deletes. What does that mean?

0 of 3 right so far. Get all three to finish this topic.

Go deeper