Question 42 of 50
How would you avoid duplicate processing?
Question
"In practice, how would you avoid duplicate message processing in a Kafka consumer?"
What the interviewer wants to assess
The most applied question in this entire section — wants to see the real implementation pattern, with specific attention to atomicity under concurrency, not just "I use a unique ID."
Resposta rápida
I use a unique eventId per event (generated by the producer), and in the consumer, I check and insert
that eventId into a control table (processed_events) with a uniqueness constraint — in the same
transaction that applies the business effect. If the insert fails due to duplication, I safely skip the
event.
Resposta nível Sênior
The critical point is atomicity: checking "have I already processed this eventId?" and applying the effect
(credit, debit, write) need to happen in the same database transaction. If they're separate steps — a
query followed by an update — there's a race window where two concurrent executions can both check "doesn't
exist" at the same time and apply the effect twice before either one writes the eventId. The database's
uniqueness constraint, not a check in application code, is the real guarantee. For operations that are
already naturally idempotent (an upsert that always writes the same final value), this explicit control can
be skipped — but for operations that accumulate state (credit, increment), it's mandatory.
In-depth explanation
See "The pattern: eventId + uniqueness constraint" and "Atomicity" in Chapter 11.
Exemplo financeiro
@Transactional
public void creditarPix(PixRecebidoEvent evento) {
try {
eventosProcessadosRepository.insert(evento.getEventId());
} catch (DataIntegrityViolationException e) {
return; // already processed
}
contaRepository.creditar(evento.getContaId(), evento.getValor());
}
"Just check whether the ID already exists before processing"
An upfront check that isn't in the same transaction as the effect is vulnerable to a race condition. The complete answer always mentions atomicity via transaction — not just the existence of a unique identifier.
Pode vir a seguir
Likely follow-ups: "what if two consumers process the same event at the same time?" and "how does this relate to the Outbox Pattern?".
Related chapters