Part IV — Reliability
Transactions and the Outbox Pattern
The dual write problem between a database and Kafka, and how the transactional outbox solves it with guaranteed consistency.
On this page
Question 10 already raised the problem: what happens if the database write succeeds but the Kafka publish fails (or vice versa)? This chapter closes out Part IV by answering that — and shows why the answer isn't "use a distributed transaction."
The Dual Write problem
Dual Write
Dual write is the (problematic) pattern of writing to two different systems — a database and Kafka — as two separate operations that aren't atomic with each other. If one succeeds and the other fails, the two systems end up inconsistent.
A typical pagamento-service, when approving a payment, does two things: writes the result to the database
(the pagamentos table) and publishes a PagamentoAprovado event to Kafka. If those two writes aren't
atomic, there are two failure windows:
- The database is written, but the Kafka publish fails (network, broker unavailable) — the payment exists, but no downstream system (notification, accounting, antifraud) ever finds out.
- The event is published to Kafka, but the database write fails afterward — downstream systems react to a payment that, in reality, was never successfully persisted.
Kafka and relational databases don't share a native transaction
There's no simple, native distributed transaction between a relational database and Kafka (XA/2PC between the two is technically possible in theory, but practically unused in production due to operational complexity and performance impact). Naively done dual write is a guaranteed source of inconsistency in production.
Transactional Outbox: the solution
Transactional Outbox
Transactional outbox is the pattern that solves dual write by writing the intent to publish an event to an
auxiliary table (outbox) in the same database transaction that writes the business effect. A separate
process — the relay — reads that table and publishes the events to Kafka asynchronously and reliably.
The core idea: since the business table and the outbox table are written in the same relational transaction (something the database already guarantees natively), they never end up inconsistent with each other — either both exist, or neither does. The problem of "reliably publishing to Kafka" turns into "read a table and publish," which is simpler to make robust (with retry, at-least-once) than coordinating two heterogeneous writes.
How the relay reads the outbox table
Two common approaches:
Relay strategies for the outbox table
| Strategy | How it works |
|---|---|
| Polling | A process periodically queries the outbox table for unpublished records, publishes to Kafka, and marks them as published |
| CDC (Change Data Capture) | A tool like Debezium reads the database's own transaction log (WAL/binlog) and automatically publishes any insert into the outbox table, with no polling |
Dica de entrevista
CDC via Debezium is considered the more robust approach because it reads the database's transaction log directly — it doesn't depend on a delayed periodic query, and it captures the event at the exact moment the transaction is confirmed. Mentioning this shows knowledge beyond the pattern itself.
Outbox guarantees publication — it doesn't eliminate duplication
"The outbox pattern solves everything, including duplication"
Outbox guarantees the event will be published if the business transaction was confirmed — it solves the lost-publication problem (the "database written, Kafka not published" scenario). It doesn't prevent the same event from being published more than once (for example, if the relay fails after publishing but before marking it as published, and retries). The consumer on the other side still needs to be idempotent (Chapter 11) — outbox and idempotency solve complementary problems, they don't replace each other.
When to use it
- Whenever a business operation needs to reliably both persist state and notify other systems via an event — the most common case in event-driven architectures with a relational database.
- Especially important when the (silent) absence of the event would be worse than a duplicated event — which is the case for most financial systems.
When it's not worth the cost
For operations where publishing the event isn't critical (occasionally losing it is acceptable, e.g., a low-value analytics event), the operational cost of maintaining an outbox table and a relay may not be justified compared to a simple direct publish, accepting the rare inconsistency window.
Relation to Java and Spring Boot
In Spring Boot, the most common implementation writes to the outbox table using the same
EntityManager/JPA transaction as the business operation, inside the same @Transactional method. The
relay is usually a separate process — either a simple scheduler (@Scheduled) doing polling, or a Debezium
connector configured to monitor the outbox table and automatically publish to Kafka, with no additional
application code.
Payment approval with outbox
The pagamento-service writes, in the same transaction, the row in pagamentos (status = approved) and a
row in outbox with the PagamentoAprovado event payload. If the transaction fails for any reason, neither
is persisted — there's no scenario where the payment is approved without the corresponding event eventually
being published, nor the reverse. Debezium, monitoring the outbox table via CDC, publishes the event to
Kafka as soon as the transaction is confirmed.
Resumo
Dual write — writing to a database and Kafka as separate operations — is a structural source of inconsistency, because the two writes don't share a native transaction. Transactional outbox solves this by writing the event's intent in the same transaction as the business operation, delegating the actual publish to a relay process (polling or CDC via Debezium). Outbox guarantees the event will be published, but doesn't eliminate the possibility of duplication — the consumer still needs to be idempotent.
Pode vir a seguir
Likely follow-ups: "why not use a distributed transaction (2PC) between the database and Kafka?" and "how would you monitor whether the outbox table's relay is lagging or stuck?" (Chapter 15).