Part IV — Reliability
Idempotency
Why consumers need to be idempotent, and how to implement it with eventId, a uniqueness constraint, and atomicity.
On this page
Chapter 10 ended by pointing here: At Least Once — the guarantee most used in financial systems — accepts duplication as a structural possibility. This chapter shows how a consumer absorbs that duplication without producing duplicated side effects.
What idempotency is
Idempotency
A process is idempotent when it produces the same final effect regardless of whether it runs once or multiple times with the same input. For a Kafka consumer, that means: processing the same event twice shouldn't produce double the effect — the balance is credited once, the email is sent once, the record exists once.
Why consumers need to be idempotent
Duplication is not a remote possibility
At Least Once (Chapter 10) guarantees no message is lost, accepting as the price that it may be delivered more than once — after a rebalance (Chapter 6), a commit retry, or a crash between finishing processing and committing the offset (Chapter 7). This isn't a rare bug: it's the expected, documented behavior of that guarantee. A consumer that doesn't handle this will duplicate effects in production, sooner or later.
The pattern: eventId + uniqueness constraint
The most common and most robust pattern: each event carries a unique identifier (eventId, usually a UUID
generated by the producer when the event is created). The consumer keeps a table (processed_events, or
similar) with a uniqueness constraint on that eventId. Before applying the business effect, the consumer
tries to insert the eventId into that table; if the insert fails due to a uniqueness violation, the event
has already been processed and the operation is safely skipped.
Atomicity: the part that's usually forgotten
"I check whether the eventId already exists before processing"
Checking and processing as separate steps (a SELECT query, then a business INSERT/UPDATE) introduces a
race condition: under concurrency (two consumers, or simultaneous reprocessing), both can check "doesn't
exist" at the same time and apply the effect twice before either one writes the eventId. The check and the
effect need to happen in the same database transaction, with the uniqueness constraint as the real
guarantee — not an upfront check in application code.
In practice, this means: inserting the eventId into the control table and applying the business effect
(debit, credit, write) in the same transaction. If the uniqueness constraint rejects the insert, the entire
transaction fails and is rolled back — no partial effect is applied, even under concurrency.
eventId vs. transactionId
Two different deduplication keys
| Key | Identifies | Typical use |
|---|---|---|
eventId | The specific Kafka event (one publish) | Deduplicating redelivery of the same event (retry, rebalance) |
transactionId (or business ID) | The domain transaction (e.g., the PIX, the payment) | Deduplicating the business operation, even if it arrives via different events |
Normally eventId is enough to handle Kafka redelivery. But in scenarios where the same business fact can
arrive through different paths (e.g., a bank notification duplicated at the source channel, producing two
distinct Kafka events for the same real transaction), deduplication by eventId alone isn't enough — a
unique business identifier is also needed (transactionId, endToEndId in the case of PIX) to catch that
duplication at the source, even before it reaches Kafka.
When natural idempotency makes the control table unnecessary
Not every consumer needs the processed_events table. If the operation is already naturally idempotent — an
upsert that always writes the same final value, regardless of how many times it runs — explicit eventId
control is redundant.
Credit score upsert
The risco-service writes each customer's latest score with an UPDATE ... WHERE clienteId = ? (upsert).
Processing the same event twice simply writes the same value twice — with no additional side effect. This
consumer is naturally idempotent and doesn't need eventId control.
Operations that increment or accumulate state (crediting a balance, incrementing a counter, sending
a notification), on the other hand, aren't naturally idempotent — processing them twice duplicates the
effect, which is why they require explicit eventId control.
How this shows up in interviews
"How would you avoid duplicate processing?" is almost guaranteed whenever At Least Once comes up in conversation. The weak answer just mentions "a unique eventId." The strong answer explains atomicity: check and apply the effect in the same transaction, with the uniqueness constraint as the real guarantee under concurrency — not an upfront check in code.
Dica de entrevista
When describing idempotency, always mention the uniqueness constraint and transaction atomicity. That
detail — not the mere existence of an eventId — is what separates a correct implementation from one that
still has a race condition under concurrency.
Relation to Java and Spring Boot
In Spring Boot, this typically translates into a table with an event_id column marked UNIQUE, and the
processing code inside a method annotated with @Transactional, which tries to insert the control record
and apply the business effect before any commit. If the insert throws a DataIntegrityViolationException
(constraint violation), the code catches that exception specifically and treats it as "already processed" —
not as a generic error to be propagated to the retry mechanism (Chapter 9).
Idempotent PIX credit
@Transactional
public void creditarPix(PixRecebidoEvent evento) {
try {
eventosProcessadosRepository.insert(evento.getEventId());
} catch (DataIntegrityViolationException e) {
return; // already processed, nothing to do
}
contaRepository.creditar(evento.getContaId(), evento.getValor());
}
Inserting the eventId and crediting the balance happen in the same transaction — if the insert fails due
to duplication, the credit is never applied a second time.
Resumo
Idempotency is what lets a consumer absorb At Least Once's structural duplication without duplicating
business effects. The central pattern is a unique eventId, checked and inserted in the same transaction
that applies the effect — the database's uniqueness constraint, not an upfront check in code, is what
guarantees atomicity under concurrency. Naturally idempotent operations (upsert) can skip this control;
operations that accumulate state (credit, increment) cannot.
Pode vir a seguir
Likely follow-ups: "what would you do if the same business event arrived under two different eventIds?"
and "how does the transactional outbox relate to idempotency?" (Chapter 12).