Integrating with an unstable external system
Timeout, retry with backoff, circuit breaker, and DLQ protecting a consumer that depends on an unreliable external API.
The previous case studies dealt with internal failures (bugs, databases, indexes). This one covers a different, common problem: a consumer that needs to call an external system — outside the team's control — that's slow or unstable, and how to prevent that external instability from bringing down the consumer's internal processing.
The scenario
The antifraude-service consumes the compras.autorizadas topic and, for each purchase, calls an external
risk-scoring API (a credit bureau or a third-party antifraud provider) before deciding whether to approve
or flag the transaction. That external provider has known instability: latency spikes and occasional
unavailability, outside the team's control.
The problem without protection
Without timeout and circuit breaker, an external failure becomes an internal problem
Without a configured timeout, a slow call to the external provider holds the consumer's thread
indefinitely — and without a circuit breaker, every message processed during a provider instability tries
the same call that's doomed to be slow, multiplying the problem instead of isolating it. The result, without
these protections: the antifraude-service quickly accumulates consumer lag, and the effect propagates as
if it were a problem with Kafka or the consumer itself — when the root cause is entirely external.
Timeout: the first layer of protection
Every call to the external provider has a short, explicit timeout (for example, 2 seconds) — if the provider doesn't respond within that window, the call is treated as a failure, freeing the consumer's thread to keep processing (or retry, depending on the retry strategy).
Circuit Breaker: the second layer
A circuit breaker prevents the consumer from continuing to call a provider that's already shown itself to be unavailable: after a number of consecutive failures, the circuit opens and starts rejecting calls immediately, without even attempting the network call — saving time and avoiding amplifying the external provider's instability with more traffic. After a configured interval, the circuit goes half-open and tests a single call: if it succeeds, the circuit closes and returns to normal; if it fails, it opens again.
Dica de entrevista
Circuit breaker and retry solve different, complementary problems: retry handles a one-off, transient failure (a specific request failed); circuit breaker handles sustained instability (the provider has been down for a while), preventing the consumer from insisting on calls doomed to fail. Mentioning this difference shows understanding beyond the isolated pattern.
Retry, backoff, and DLQ
When the circuit is closed but a call fails on its own (timeout or a 5xx error from the provider), the
consumer treats it as a transient error (Chapter 9): retry with exponential backoff, a few attempts. If the
circuit is open (rejecting outright) or the attempts run out, the message goes to a specific DLQ —
compras.autorizadas.antifraude.dlq — for manual or automatic reprocessing as soon as the external
provider recovers.
Manual reprocessing
"Messages in the DLQ due to external instability should be discarded"
Unlike a DLQ populated by permanent errors (malformed payload), messages in the DLQ due to an external provider's instability generally should be reprocessed as soon as the provider recovers — the error wasn't with the data, it was the momentary availability of a third-party system. A process (manual or automated) monitors the provider's health and, once it responds normally again, republishes the DLQ messages back into the main flow.
Idempotency is still necessary
Even with all these protections, the consumer still needs to be idempotent (Chapter 11): if a call to the
external provider succeeds, but the consumer crashes before committing the offset, the message will be
reprocessed — and the call to the external provider (which may have a per-call financial cost, in the case
of credit bureaus) shouldn't be redone unnecessarily. A result-cache table keyed by eventId avoids both
duplicated effects and the unnecessary cost of repeated calls to the provider.
Observability
Signals that distinguish an internal problem from an external one
| Signal | Points to |
|---|---|
| High consumer lag + circuit breaker open | The problem is the external provider, not the consumer |
| High consumer lag + circuit breaker closed + low error rate | The problem is internal processing capacity |
| DLQ message rate correlated with known provider downtime | Confirms an external cause, directing the response toward reprocessing instead of code investigation |
Resumo
Consumers that depend on unstable external systems need specific protection layers: timeout to avoid holding threads indefinitely, circuit breaker to stop insisting on a provider already identified as down, and retry with backoff for the one-off failures still worth trying again. DLQ isolates what couldn't be processed during the instability, for reprocessing as soon as the external provider recovers — and idempotency remains mandatory, both for correctness and, in this case, for cost.
Related chapters