Part II — Architecture
Internal architecture
Producer, Broker, Cluster, Topic, Partition, Offset, Consumer, and Consumer Group — the pieces that make up Kafka.
On this page
This chapter precisely defines the pieces that will show up in practically the rest of the book. If Chapter 1 answered "why does Kafka exist," this chapter answers "what is it made of."
Producer
Producer
A Producer is any application that publishes events to a Kafka topic. In practice, it's a microservice (or part of one) that, at the end of a business operation, sends a message representing the fact that just occurred.
A Producer decides which topic to send to, which key to use (Chapter 4), and how to serialize the
value (JSON, Avro, Protobuf). It doesn't know — and shouldn't need to know — how many consumers exist or
what they do with the event. In Spring Boot, a Producer is typically a @Service using
KafkaTemplate<K, V> (Chapter 13).
Broker
Broker
A Broker is a running Kafka process, responsible for storing data for a subset of the cluster's partitions and serving read and write requests for those partitions.
A single broker is already a functional Kafka, but in production it always runs alongside other brokers, forming a cluster, for fault tolerance and load distribution.
Cluster
Cluster
A Cluster is the set of brokers that together store and serve all topics in a Kafka installation. Coordination between brokers (controller election, cluster metadata) is done via Kafka Raft (KRaft) in current versions, replacing the old ZooKeeper.
Topic
Topic
A Topic is the logical name under which related events are published and consumed — for
example, pagamentos.aprovados or pix.recebido. A topic is an abstraction; physically, it's
split into one or more partitions.
A topic has no schema imposed by Kafka itself (the schema, when it exists, is the responsibility of a layer like the Schema Registry, outside this chapter's scope). What Kafka guarantees is delivery order within each partition of the topic — not across different topics, and not necessarily across partitions of the same topic.
Partition
Partition
A Partition is an ordered, immutable subdivision of a topic — the actual physical log. Each partition is a sequence of records identified by an increasing offset, stored on a broker (with replicas on other brokers, see Chapter 5).
Partitions exist for two reasons: parallelism (multiple consumers can process different partitions at the same time) and horizontal scalability (the topic can grow by spreading partitions across more brokers). Chapter 4 dives into how the key choice determines which partition a message lands on, and why that's the most important design decision when modeling a topic.
Offset
Offset
An Offset is the sequential position of a record within a partition — an increasing integer, unique per partition, that uniquely identifies a message in that partition.
Offset isn't global to the topic; it's local to each partition. The same offset number exists independently in each partition of a topic. Chapter 7 details how consumers use the offset to know "how far I've already read."
Consumer
Consumer
A Consumer is any application that reads events from one or more topics, processing them from the position (offset) where it last stopped.
A Consumer, on its own, reads from every partition it's assigned to. In Spring Boot, the pattern is a
method annotated with @KafkaListener (Chapter 14).
Consumer Group
Consumer Group
A Consumer Group is a set of consumer instances identifying with the same group.id, splitting
a topic's partitions among themselves — each partition is read by exactly one consumer in the
group at a given time.
It's the Consumer Group that turns Kafka into a tool capable of both distributing work (within a group, each partition goes to a single consumer, like in a queue) and sharing the same data with multiple systems (different groups read the same topic completely independently, each with its own set of offsets). Chapter 6 details rebalance behavior when consumers join or leave a group.
Putting the pieces together
The full flow: a Producer publishes to the pagamentos topic; Kafka decides (via key/hash, Chapter 4)
which of the topic's partitions the message lands on; it's persisted with a sequential offset in that
partition, replicated to the replica brokers (Chapter 5); one or more Consumer Groups read the partition
independently, each group advancing its own set of offsets.
When this model is the right choice
This design pays off when the number of events and independent consumers justifies the complexity — typically starting from a handful of microservices reacting to the same event domain. For a single producer/consumer with low volume, the same architecture technically works, but the operational cost of maintaining partitions, replicas, and a cluster rarely pays off.
Limitations to keep in mind
- A topic's number of partitions defines the maximum consumption parallelism within a Consumer Group — increasing it later is possible, but redistributes the keys (Chapter 4).
- More partitions isn't always better: each partition consumes broker resources (file handles, memory) and increases rebalance time.
How this shows up in interviews
Interviewers often ask you to draw this flow on a whiteboard (or describe it verbally) and then vary the parameters: "what changes if I have 2 partitions and 5 consumers in the group?", "what if I have two different Consumer Groups reading the same topic?". Answering precisely that the offset is local to the partition — not the topic — is a signal often used to tell apart someone who's just used Kafka from someone who understands it.
Dica de entrevista
When describing the architecture, make it explicit that the offset is per partition, not per topic, and that different Consumer Groups are completely independent of each other — two groups reading the same topic don't compete for the same messages.
Relation to Java and Spring Boot
Each piece maps to a Spring Kafka configuration or class: bootstrap-servers points to the cluster's
brokers; KafkaTemplate.send(topic, key, value) is the Producer; @KafkaListener(topics = "...", groupId = "...") defines the Consumer and its Consumer Group; the offset is managed by the Acknowledgment (manual
commit) or automatically by the client, depending on configuration (Chapter 7).
Card statements
The faturas.fechadas topic might have 12 partitions, a key strategy by accountId (to
guarantee that all events for the same account stay ordered on the same partition), and two
Consumer Groups: one for the billing service (which generates the invoice) and another for the
analytics service (which feeds a dashboard) — both read the same topic, completely
independently, each at its own pace.
Resumo
Producer publishes, Broker stores, Cluster is the set of brokers, Topic is the logical name, Partition is the ordered physical log, Offset is the position within the partition, Consumer reads, and Consumer Group is the mechanism that lets you both distribute work and share the same data across independent systems.
Pode vir a seguir
Common follow-ups: "why do partitions exist?", "how does Kafka choose a message's partition?", and "what happens if I have more consumers than partitions?".