Part I — Fundamentals

What is Apache Kafka?

Origin, context, the problem Kafka solves, and why it isn't just a queue.

On this page

Before getting into Producer, Broker, or Partition, it's worth answering a question most developers never stop to answer precisely: why does Kafka exist? If you can't justify why a piece of infrastructure exists, you also can't justify when to use it — and that's exactly the kind of judgment a senior interview tries to measure.

The problem before Kafka

In a system with few services, point-to-point integration works fine: the payments service calls the notification service directly over HTTP, which calls the antifraud service, and so on. The problem shows up when the number of services grows. Every new consumer of a payment event means a new synchronous call, a new point of failure, and a new direct coupling between teams.

Picture a "payment approved" event in a digital bank. Today it might interest: push notifications, antifraud, cashback, accounting, analytics, and credit limit updates. If the payments service calls each of these synchronously over HTTP, it becomes dependent on the availability of all of them — and adding a seventh consumer requires changing the payments service's code again.

It's not only about performance

The core problem isn't request volume — it's coupling. A system where the producer of an event needs to know all of its consumers doesn't scale organizationally, even if it scales in throughput.

Event streaming and event-driven architecture

Kafka was born at LinkedIn, around 2010, to solve the problem of moving large volumes of activity data (clicks, views, metrics) between systems reliably and decoupled. The core idea is simple to state and deep in practice: instead of systems calling each other directly, they publish and consume events through a shared distributed log.

Event streaming

Event streaming is the architectural pattern in which facts that already happened ("payment approved", "invoice paid", "card blocked") are published as immutable events to a central log, and any number of interested systems can consume them independently, at their own pace.

This is different from simply "sending a message." An event isn't a command ("process this invoice"); it's a statement about something that already happened. The producer doesn't know — and doesn't need to know — who will consume the event or what each consumer will do with it. That inversion is the foundation of event-driven architecture: the payments system publishes "payment approved" and moves on; antifraud, notifications, and accounting discover that fact asynchronously, each at its own pace.

Kafka as a distributed log

Technically, Kafka is implemented as a distributed, partitioned, and replicated log. "Log" here isn't an application log — it's the data structure: an ordered, immutable sequence of records, each identified by an increasing position (the offset). New events are always appended at the end; nothing is modified in place.

A Producer publishes to a Topic; the Topic is split into Partitions; multiple Consumers read independently.

This log structure is what enables two properties that make Kafka fundamentally different from a traditional queue:

Multiple independent consumers can read the same set of events, each keeping its own read position (offset), without one consumer's reading affecting another's.
Configurable retention means the event stays available in the log after being consumed — a new service can be born tomorrow and read the full "payment approved" history from the last 7 days without the payments system needing to resend anything.

Kafka is not just a queue

This is probably the most repeated — and most incomplete — comparison about Kafka.

"Kafka is just a faster queue"

This statement ignores the most important structural difference: in a classic queue (RabbitMQ, SQS), a consumed message normally leaves the queue — it exists to be delivered once and disappear. In Kafka, consuming an event doesn't remove it from the log; it stays there until the retention policy expires it, regardless of how many consumers have already read it.

A queue models work to be distributed — each item should be processed by exactly one worker. Kafka models facts to be shared — the same event can (and usually should) be read by several different systems, each with its own interpretation of the fact. It's possible to use Kafka to distribute work (that's what a Consumer Group does within a partition), but reducing Kafka to that misses half the reason it was designed the way it was.

When to use it

Multiple systems need to react to the same business fact, independently.
You need replay: reprocessing old events to fix a bug, populate a new system, or rebuild state (e.g., reindexing an Elasticsearch cluster from an event history).
Volume and event rate are high enough that synchronous coupling becomes an availability risk (one service being down shouldn't block the others).
You want to decouple producers from consumers at the organizational level — different teams evolving their services without coordinating deploys.

When not to use it

Strictly request/response communication, where the caller needs an immediate answer (e.g., authorizing a card transaction end-to-end in real time) — that's the job of a synchronous call (HTTP/gRPC), not an asynchronous event.
Small systems, with few services and low volume, where the operational complexity of a Kafka cluster doesn't pay off. A simple queue or even direct calls solve it with less effort.
When the team doesn't have — and isn't going to build — the minimum operational maturity to run a stateful distributed system (lag monitoring, partition management, cluster upgrades).

Limitations

Kafka does not guarantee global ordering across partitions — only within a single partition (we come back to this in Chapter 4).
It's not a transactional database; although it supports transactions (Chapter 12), it doesn't replace a relational database for ad-hoc queries over current state.
It has a real operational learning curve: poorly sized partitions, misunderstood rebalances, and misconfigured retention are common causes of production incidents.

How this shows up in interviews

Practically every Kafka interview starts with some variation of "what is Kafka" or "why use Kafka instead of a queue." The interviewer doesn't want the dictionary definition — they want to see whether you understand the coupling problem that motivated the tool's creation, and whether you can tell a "work queue" apart from a "shared event log." Answers that only cite "high performance" or "processes lots of data" sound rehearsed and rarely convince.

Dica de entrevista

Structure the answer in three steps: (1) the problem — coupling between a producer and multiple consumers; (2) the solution — a distributed, immutable log with multiple independent consumers; (3) the consequence — replay, organizational decoupling, and the fundamental difference from a queue.

Relation to Java and Spring Boot

In practice, a Java developer interacts with Kafka through the official client (kafka-clients) or, more commonly, via Spring Kafka, which wraps KafkaTemplate for producing and @KafkaListener for consuming (Chapters 13 and 14). The idea of "publish an event and move on" translates directly into code: a payments @Service publishes an event via KafkaTemplate.send(...), ideally within the same transaction that persists the payment to the database, and returns without waiting for antifraud, notifications, or accounting.

How this shows up in real systems

PIX received, without Kafka vs. with Kafka

Without an event log, the service that receives a PIX confirmation would need to call the balance service, the push notification service, the statement service, and the antifraud service directly — either synchronously or with a dedicated queue for each. With Kafka, it publishes a single PixRecebido event to a topic; balance, notifications, statement, and antifraud consume that same event independently, each in its own Consumer Group, without the PIX-receiving service knowing they exist.

Resumo

Kafka solves the coupling problem between an event producer and multiple independent consumers, through a distributed, partitioned, and replicated log where events are appended and retained for a configurable period — not removed once consumed. That makes it structurally different from a traditional queue, which models work distribution, not fact sharing.

Pode vir a seguir

After answering "what is Kafka," be ready for: "is Kafka a queue?", "when would you not use Kafka?", and "what real problem have you solved with Kafka?".