sdlc.dev

Kafka path · Writing and reading · Topic 5 of 12

Delivery semantics and exactly-once

Not started

Every messaging system has to answer one question: when something fails halfway, is a message lost, delivered twice, or neither? Kafka's design documentation lists three answers. At most once: records may be lost, never redelivered. At least once: never lost, sometimes redelivered. Exactly once: each record's effect happens once. Kafka gives at-least-once by default, and exactly-once for pipelines that read from Kafka and write back to Kafka.

Why it matters: "Exactly-once" is the most misunderstood phrase in Kafka. It is real, but it covers reading from Kafka, processing and writing back to Kafka, not the email your consumer sends or the row it writes to PostgreSQL.

  1. 1Three possible guarantees

    At-most-once (commit before processing), at-least-once (commit after processing, the default) and exactly-once (the result appears once even if steps are retried).

  2. 2Transactions tie output to input

    A transactional producer writes output records and the consumer's offsets in one atomic transaction, so either both happen or neither.

  3. 3Readers must ask for committed data

    Consumers see aborted and in-flight transactional records unless isolation.level=read_committed; the default is read_uncommitted.

  4. 4It stops at Kafka's edge

    Side effects outside Kafka (databases, emails, HTTP calls) are not covered. They need idempotent handlers or the outbox and inbox patterns.

1

How it works

The explanation, with examples you can run.

Where duplicates and gaps come from

  • Producer side. A send is retried after a lost acknowledgement. The idempotent producer (default since 3.0) removes these duplicates within a producer session.
  • Consumer side. The consumer processes records, then crashes before committing. The next owner starts from the old commit and processes them again (the trace in the previous topic). Committing before processing turns this into a gap instead.

So, by default: no loss with acks=all and sensible replication, but possible duplicates at the consumer.

Exactly-once: read, process, write

For a consumer that reads topic A and writes results to topic B, the dangerous moment is between writing to B and committing A's offset. Kafka's transactions put both in one atomic unit:

# producertransactional.id=payments-enricher-1enable.idempotence=true# consumerisolation.level=read_committedenable.auto.commit=false
producer.initTransactions();while (true) {  ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));  producer.beginTransaction();  for (var r : records) producer.send(new ProducerRecord<>("payments-enriched", r.key(), enrich(r.value())));  producer.sendOffsetsToTransaction(nextOffsets(records), consumer.groupMetadata());  producer.commitTransaction();   // output and input offsets together, or neither}

(Error handling omitted: on failure, abort and rewind.) The transactional.id survives restarts, so a restarted instance fences off its old self and the old instance can no longer commit. Consumers with read_committed only see records from committed transactions; the default read_uncommitted also returns aborted ones. Kafka Streams does all of this for you with processing.guarantee=exactly_once_v2.

Kafka 4.0 strengthened the protocol on the broker side (KIP-890): with 4.0 clients the producer epoch is bumped on every transaction, so a late, retried write cannot slip into the next transaction.

What it does not cover

Exactly-once covers records in Kafka and offsets in Kafka. It does not cover:

  • writes to a database, a cache or a search index;
  • emails, payments, HTTP calls;
  • a producer that calls send() twice because your code ran twice.

For those, design the consumer to be idempotent: upsert by a business key, keep a table of processed record ids (an inbox) in the same database transaction as the effect, or carry an idempotency key to the external API. With Kafka Connect, whether a sink is exactly-once depends on the connector and the target system, so read its documentation. The pragmatic default for most services is at-least-once delivery plus idempotent handling.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.A Kafka Streams app with processing.guarantee=exactly_once_v2 reads payments and also calls an external fraud API per record. Is the API called exactly once per record?
  2. 2.A transactional producer aborts a transaction. A consumer with the default isolation.level reads the topic. What does it see?
  3. 3.Which setup gives at-most-once delivery?

0 of 3 right so far. Get all three to finish this topic.

Go deeper