sdlc.dev

Kafka path · The log · Topic 2 of 12

Partitions, keys and ordering

Not started

A topic is divided into partitions, each an independent log on some broker. Partitions let a topic grow beyond one machine and let many consumers read in parallel. The price is that Kafka only orders records within a partition. If two records must be processed in order, they must be in the same partition, which in practice means they must have the same key.

Why it matters: Partitions are how Kafka scales, and they are also the limit of its ordering guarantee. The key you choose decides which events stay in order and whether load spreads evenly.

  1. 1Ordering holds only within a partition

    Records in one partition are read in the order they were written. Across partitions there is no order at all.

  2. 2The key picks the partition

    By default the producer hashes the key (murmur2) modulo the partition count, so every record with the same key lands in the same partition.

  3. 3No key means spread

    Records without a key are spread by a sticky partitioner, which fills a batch for one partition before moving to another.

  4. 4Adding partitions moves keys

    Changing the partition count changes hash mod n, so new records for a key may go to a different partition than its old ones.

1

How it works

The explanation, with examples you can run.

Keys pick partitions

Producing six order events with keys:

printf "customer-42:created\ncustomer-7:created\ncustomer-42:paid\ncustomer-99:created\ncustomer-7:paid\ncustomer-42:shipped\n" \  | bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders \      --property parse.key=true --property key.separator=:

Reading them back with print.partition and print.offset:

Partition:0	Offset:0	customer-42	createdPartition:0	Offset:1	customer-42	paidPartition:0	Offset:2	customer-42	shippedPartition:1	Offset:0	customer-7	createdPartition:1	Offset:1	customer-99	createdPartition:1	Offset:2	customer-7	paid

All of customer-42's events are in partition 0, in order. customer-7 and customer-99 share partition 1, interleaved; that is fine, because order only matters per key. Partition 2 got nothing: with three keys, an even spread is not guaranteed.

The default partitioner, in JavaScript

For a record with a key, the Java producer takes the murmur2 hash of the key bytes, clears the sign bit and takes it modulo the number of partitions. This port gives the same answers the cluster gave above (and, for 4 partitions, the same as a second test topic):

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

Note the last column: going from 3 to 4 partitions moves every one of these keys. Kafka does not move existing records, so after adding partitions a key's old records are in one partition and its new ones in another, and a consumer can see them out of order across the switch. For keyed topics, choose a partition count with room to grow and avoid changing it.

Records with no key are spread for throughput. The Java producer's default logic sends keyless records to one partition until batch.size bytes have gone there, then switches, which builds fuller batches than round-robin. Clients in other languages (librdkafka, for example) may use a different hash by default, so a mixed-language system should agree on a partitioner if keys must line up.

Choosing a key

  • Use the entity whose events must stay in order: order id, account id, device id.
  • Check the spread. One key with a large share of traffic makes one hot partition and one overloaded consumer.
  • Partition count caps consumer parallelism in a group: with 12 partitions, a 13th consumer sits idle.
2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Order events (created, paid, shipped) must be processed in order per order. Which key should the producer use?
  2. 2.You increase a topic from 3 to 4 partitions. What happens to the key customer-42?
  3. 3.One tenant produces 60% of all traffic, and the topic is keyed by tenant id. What is the problem?

0 of 3 right so far. Get all three to finish this topic.

Go deeper