Part III — Consumption and reprocessing

Offset and Commit

How a consumer marks how far it has read, the difference between auto commit and manual commit, and the risks of each approach.

On this page

Previous chapters mentioned offset as "a record's position in the log." This chapter goes into the detail that shows up most often in production incidents: how and when that position is saved — and what happens when it's saved at the wrong moment.

Offset as read position

Offset

Offset is an increasing integer, unique per partition, that identifies a record's position in the log. For a consumer, the offset also represents "how far I've already read" — but that position is only officially recorded when the consumer commits it.

It's essential to separate two concepts that look like the same thing: the consumer's in-memory read position (where the client has already fetched and delivered messages to the application) and the committed offset (what's persisted in Kafka as "the Consumer Group has processed up to here"). Between the two there's a window where messages have already been delivered to the application but aren't yet officially marked as processed.

Offsets 0-3 already committed; offsets 4-6 already processed by the application but not yet committed; offset 7 not yet read.

Offset is per partition and per Consumer Group

Two dimensions that can't be confused

The committed offset isn't a global property of the topic or of the consumer in isolation — it's stored per combination of partition and Consumer Group. That means two different Consumer Groups reading the same topic each have their own set of committed offsets, completely independent of each other.

That's why a new Consumer Group, when it starts consuming an existing topic, has no committed offset yet — and the auto.offset.reset setting (earliest or latest) decides whether it starts from the beginning of the available retention window or only from new events going forward.

Auto Commit

With enable.auto.commit=true (the default), the Kafka client automatically commits processed offsets on a periodic interval (auto.commit.interval.ms), without the application needing to call anything explicitly.

The advantage is simplicity: the developer writes no commit-related code. The risk is that the commit happens on a time interval, not in sync with the actual message processing — it's perfectly possible for the client to commit an offset whose processing by the application is still in progress, or worse, that failed silently.

Manual Commit

With enable.auto.commit=false, the application explicitly decides when to commit — typically right after confirming that processing of a message (or batch) finished successfully, via Acknowledgment.acknowledge() in Spring Kafka or consumer.commitSync()/commitAsync() in the plain client.

Manual commit gives precise control over the relationship between "I processed it" and "I committed it," at the cost of more code and more developer responsibility for getting that coupling right.

The risks of each approach

Auto commit vs. manual commit: what can go wrong

Order	Risk
Commit before processing finishes	If the consumer crashes mid-processing, the message is considered processed even though it wasn't — message loss
Processing finishes before the commit	If the consumer crashes before the commit happens, the message will be redelivered on the next startup — duplication

"Auto commit is safer because it is automatic"

Automatic doesn't mean safe — it means the commit's timing is outside the application's control. Auto commit tends toward the message loss scenario: it can commit an offset whose processing hasn't finished yet (or failed), because the commit interval knows nothing about the real processing state.

Manual commit, done after processing finishes successfully, eliminates the risk of message loss, but doesn't eliminate duplication: if the consumer crashes exactly between "I finished processing" and "I committed," the message will be reprocessed. That's why at-least-once (Chapter 10) requires idempotent processing (Chapter 11) — duplication isn't a bug to be eliminated, it's a structural possibility the consumer needs to know how to absorb.

When this turns into a real incident

Teams that use auto commit in asynchronous or multi-threaded processing — where the message is delivered to the application and actual processing happens on another thread or internal queue — often discover too late that the client already committed the offset before processing actually finished. An application crash in that window silently loses the message, with no exception, no error log — because, from Kafka's point of view, it was already processed.

Dica de entrevista

When explaining the difference between auto commit and manual commit, don't stop at "one is automatic and the other manual" — explain the consequence: auto commit tends to lose messages under failure, well-positioned manual commit (after processing) tends to duplicate messages under failure. Neither is "safer" in the abstract; the correct design depends on accepting duplication and handling processing with idempotency.

Relation to Java and Spring Boot

In Spring Kafka, enable.auto.commit=false combined with AckMode.MANUAL (or MANUAL_IMMEDIATE) on the ContainerFactory is the recommended default for processing with real guarantees: the method annotated with @KafkaListener receives an Acknowledgment parameter, and calls acknowledgment.acknowledge() only after all of the message's business logic — including any database write — finished successfully.

Processing overdue invoices

The cobranca-service consumes the boletos.vencidos topic with manual commit: it only calls acknowledgment.acknowledge() after confirming the collection notification was sent and recorded in the database. If the service crashes mid-send, the offset doesn't advance — the invoice will be reprocessed on the next startup, producing at most a duplicate notification (mitigated by an idempotency check, Chapter 11), never a silently lost charge.

Resumo

Offset marks the read position within a partition; commit is what actually persists that position for a Consumer Group. Auto commit is simple, but commits on a time interval misaligned with actual processing, tending to lose messages under failure. Manual commit, done after processing is confirmed, tends to duplicate messages under failure instead of losing them — which is why idempotent processing is a requirement, not an extra, in any consumer with at-least-once guarantees.

Pode vir a seguir

Likely follow-ups: "what is auto.offset.reset and when does it kick in?", "how would you guarantee zero message loss in your consumer?", and "what is exactly-once, and does it eliminate the need for idempotency?" (Chapters 10 and 11).