Question 32 of 50
How does Kafka manage to reread consumed messages?
Question
"Technically, how does Kafka manage to reread messages that were already consumed?"
What the interviewer wants to assess
Whether you understand the real mechanism behind replay — the correct answer is about the nature of the log, not about some separate "magic" feature.
Resposta rápida
Because consuming a message doesn't remove it from the log — unlike a traditional queue, Kafka keeps the message retained (until it expires under the retention policy) regardless of whether it's been read. Rereading is just moving the Consumer Group's committed offset back to an earlier position.
Resposta nível Sênior
There's no separate "replay" mechanism in Kafka — it's a direct consequence of how the log works. A partition is an immutable sequence of records identified by offset; consuming a message only advances the consumer's read position (and, if committed, the Consumer Group's official position). The message itself stays on the broker's disk, available for any future read, until the retention policy discards it. So "rereading" is simply instructing the consumer (or a new Consumer Group) to fetch from an earlier offset — the Kafka client then goes back to reading from exactly that point, as if it had never advanced.
In-depth explanation
See "Replay: reprocessing what's already been read" in Chapter 8.
Exemplo financeiro
Rebuilding the Elasticsearch index used for transaction search is done by creating a new Consumer Group
with auto.offset.reset=earliest — it reads the entire topic from the start of the available retention,
rebuilding the index from scratch, with no special action needed on the producer side.
"Kafka keeps a separate copy to enable replay"
There's no separate copy — it's the same partition, the same log. Replay doesn't duplicate data or use a parallel mechanism; it just repositions where the consumer starts reading within the log that already exists.
Pode vir a seguir
Likely follow-ups: "what happens if I try to reread something that already expired under retention?" and "what's the difference between Replay and SQS DLQ Redrive?".
Related chapters