sdlc.dev

Kafka path · The log · Topic 1 of 12

The log: append-only, with offsets

Not started

A Kafka topic is a log: an ordered sequence of records that only grows at the end. Producers append records; each one gets an offset, its position in the log. Consumers read from an offset of their choosing and move forward, and reading changes nothing in the log. That one idea separates Kafka from a classic message queue, where a delivered and acknowledged message is gone.

Why it matters: Most of Kafka's behaviour (replay, many independent readers, ordering, retention) follows from one decision: a topic is a log that readers do not remove anything from.

  1. 1Records are appended, never edited

    Each record (key, value, headers, timestamp) is added to the end of a partition's log and gets the next offset: 0, 1, 2 and so on.

  2. 2Reading does not delete

    A consumer reads from an offset and moves forward. The record stays for every other reader until retention removes it.

  3. 3Each reader keeps its own position

    The broker does not track which messages were delivered; a consumer (or its group) stores the offset it has reached.

  4. 4Sequential and cache-friendly

    Appends and reads are sequential I/O through the OS page cache, and brokers send log data to sockets without copying it through the JVM.

1

How it works

The explanation, with examples you can run.

A topic, as the broker sees it

bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
Topic: orders  TopicId: mKocV5hnSlCzNaZDxh26kA  PartitionCount: 3  ReplicationFactor: 1  Configs: min.insync.replicas=1,segment.bytes=1073741824    Topic: orders  Partition: 0  Leader: 1  Replicas: 1  Isr: 1  ...    Topic: orders  Partition: 1  Leader: 1  Replicas: 1  Isr: 1  ...    Topic: orders  Partition: 2  Leader: 1  Replicas: 1  Isr: 1  ...

A topic is split into partitions (the next topic), and each partition is a separate log. On disk a partition is a directory of segment files:

/tmp/kraft-combined-logs/orders-0/  00000000000000000000.log        the records  00000000000000000000.index      offset -> byte position  00000000000000000000.timeindex  timestamp -> offset  leader-epoch-checkpoint  partition.metadata

Each segment is named after its first offset. When it reaches segment.bytes (1 GiB by default) or segment.ms, the broker starts a new one. Retention deletes whole old segments, which is why removing old data is cheap.

Reading by position

Any consumer can start anywhere still retained. After writing six orders, reading partition 0 from offset 1:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \  --partition 0 --offset 1 --property print.offset=true --property print.key=true --max-messages 2
Offset:1	customer-42	paidOffset:2	customer-42	shipped

The broker does not record that these were read. A consumer that wants to resume later stores its own position: consumer groups commit offsets to an internal topic, __consumer_offsets. The offset committed is the next one to read, not the last one processed.

Why it is fast

The design documentation makes two points. Appending to and reading from the end of files is sequential I/O, which disks and SSDs handle well, and Kafka leans on the operating system's page cache instead of keeping its own cache of records in the JVM heap. When a consumer fetches data, the broker can hand file data straight to the socket (sendfile on Linux) without copying it through the application. Producers and consumers also work in batches, so per-record overhead is small.

Consumers pull rather than being pushed to. A slow consumer simply falls behind and catches up later; the gap is called consumer lag, and it is the main health signal for anything reading from Kafka.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Three services read the same topic. One of them is down for an hour. What happens to the records it misses?
  2. 2.What is an offset?
  3. 3.Why can a consumer replay yesterday's data from Kafka but not from a typical work queue?

0 of 3 right so far. Get all three to finish this topic.

Go deeper