Glossary

In Kafka, ACK (acknowledgment) is the producer setting (acks) that defines how much confirmation it requires before considering a write successful: acks=0 waits for no confirmation at all (fastest, riskiest), acks=1 waits for confirmation from the leader only, and acks=all waits for confirmation from every replica in the ISR (slowest, most durable). This trade-off is central to any discussion of delivery guarantees.

A broker is a running instance of the Kafka server. It physically stores the data for one or more partitions and responds to requests from producers (writes) and consumers (reads) for those partitions. A Kafka cluster is made up of multiple brokers, each responsible for a subset of the partitions across all topics — and, for each partition, one broker acts as leader while the rest (if there are replicas) act as followers.

The cluster is Kafka's unit of deployment: a group of brokers coordinated with each other (via KRaft in current versions) that together provide fault tolerance and load distribution. Topics and their partitions are spread across the cluster's brokers; losing one broker doesn't bring down the whole cluster, as long as the affected partitions have replicas on other brokers.

Commit is the operation that records, for a Consumer Group, up to which offset of a partition has already been successfully processed. It can be automatic (enable.auto.commit=true, the client commits periodically in the background) or manual (the application decides exactly when to commit, typically after confirming that processing of the message has finished). The choice between auto commit and manual commit defines the consumer's behavior in the face of a failure: whatever was processed but not yet committed gets reprocessed from scratch after a restart or rebalance.

A consumer reads records from one or more partitions of a topic, advancing sequentially from the last processed offset. Consumers almost always operate within a Consumer Group, which determines how the topic's partitions are split across the running instances.

Within a Consumer Group, each partition of a topic is assigned to exactly one consumer in the group at any given time — which distributes work, similar to a queue. Different Consumer Groups, however, are completely independent of each other: each group keeps its own set of offsets and can read the same topic without interfering with the others. It's this combination that lets Kafka both distribute work and share the same data with multiple systems.

Consumer lag is the most important metric for knowing whether a Consumer Group is keeping up with a topic's production rate. Calculated per partition as log end offset - committed offset, it represents how many messages have already been written but not yet processed. Zero lag means the consumer is caught up; lag that grows over time indicates the consumer is processing slower than the producer is writing — a sign you need to investigate processing throughput, increase parallelism (more partitions and consumers, Chapter 6), or optimize the consumer's code.

DLQ (Dead Letter Queue, or Dead Letter Topic in Kafka's vocabulary) is a dedicated topic where messages that repeatedly failed processing are published, after retry attempts are exhausted. Kafka has no native DLQ mechanism like SQS does — it's implemented by the application (or by a framework, like Spring Kafka via DeadLetterPublishingRecoverer), publishing the problematic message to a separate topic. The main benefit is isolating the "poison pill" effect: a message that gets stuck doesn't block the rest of that partition's messages indefinitely.

An event is the fundamental unit of data in Kafka: an immutable record, published to a topic, representing a business fact that already occurred — "payment approved," "PIX received," "account blocked." Unlike a command (which instructs a future action), an event only reports something that already happened; whoever consumes it independently decides what to do with that information. It's this nature — a past fact, not an instruction — that lets multiple consumers react to the same event in completely different ways, without the producer needing to know any of them.

A follower keeps a copy of the partition data it replicates, continuously fetching new records from the leader. It doesn't serve client reads or writes directly (in the default configuration) — its role is to let an up-to-date follower take over as the new leader without losing confirmed data if the leader fails.

Idempotency, in the context of Kafka consumers, is the ability to process the same event multiple times without producing duplicated side effects — crediting a balance twice, sending two emails, generating two charges. It's a requirement, not a nice-to-have, because at-least-once guarantees (the most common setup in production) imply the same event can be delivered more than once, due to retries, rebalances, or failures between processing and committing the offset. The most common implementation uses a unique event identifier (eventId) and a database uniqueness constraint, checked and applied within the same transaction that executes the business effect.

ISR (In-Sync Replicas) is the list of a partition's replicas — including the leader itself — that are caught up with confirmed data. Only followers in the ISR are eligible for election as the new leader in case of failure, preventing a lagging replica from taking over and causing loss of data already confirmed to producers.

The key is the mechanism that guarantees relative ordering between messages: all messages with the same key land on the same partition (via hashing the key), and are therefore read in the order they were written. Messages without a key are distributed across partitions in an approximately uniform way (round-robin/sticky), with no ordering guarantee between them.

Every partition has exactly one broker acting as leader at any given time. Producers and consumers interact with the partition through the leader; the other brokers holding a replica of that partition act as followers, replicating the leader's data. If the leader fails, an up-to-date follower is elected as the new leader.

The offset is an increasing integer, unique per partition, marking a record's position in the log. It's not global to the topic — it's local to each partition; the same offset numbering exists independently in each partition. Consumers use the offset to know how far they've read, and committing the offset (automatically or manually) is what marks that progress.

A partition is the real unit of storage and parallelism in Kafka. Each partition is an append-only log: new records are always added at the end, identified by an increasing offset. Partitions exist for two reasons — parallelism (multiple consumers can process different partitions at the same time) and scalability (a topic grows by spreading its partitions across more brokers). The message's key determines which partition it's stored in.

The producer is the role played by any application — typically a microservice — that sends messages to a topic. It decides the destination topic, the message's key (which determines the partition), and the value's serialization format. A producer doesn't know its consumers; that absence of coupling is what makes event-driven architecture possible.

Rebalance is the process by which the Consumer Group coordinator redistributes a topic's partitions among the active consumers, whenever the group's membership changes — a consumer goes down, a new one joins, or a consumer is considered dead for not sending a heartbeat in time. During the rebalance, the group's consumption pauses briefly while partitions are reassigned; more recent rebalance strategies (cooperative sticky) reduce that impact by avoiding revoking partitions from consumers that remain active.

Replay is the ability to reread events already retained in the log, moving a Consumer Group's offset (or a new instance of one) to an earlier position — or to the start of the available retention window. It's used to fix processing bugs, populate a new system with existing history, or rebuild derived state (like a search index). It depends directly on the topic's retention policy: you can only reread what hasn't expired yet.

Replication factor is configured per topic and determines how many replicas of each partition exist in the cluster — one acting as leader and the rest as followers. A replication factor of 3 means the cluster can lose up to 2 brokers holding replicas of that partition without losing confirmed data, as long as the remaining replicas are in the ISR at the moment of failure. Replication factor shouldn't be confused with the number of partitions: one defines how many copies exist of each partition (availability), the other defines how many slices the topic is split into (parallelism and scale).

Retention determines when a record can be discarded from the log — by time (e.g., 7 days) or by the partition's accumulated size, whichever comes first. Unlike a queue, retention in Kafka doesn't depend on the record having already been consumed: it stays available to any consumer (existing or future) until it expires under the configured policy. Retention is what makes replay possible.

Retry is the attempt to reprocess a message after a failure, based on the premise that the error might be transient (a momentarily unavailable database, a network timeout) and that a new attempt, maybe seconds or minutes later, has a chance of succeeding. Best practice is to use exponential backoff (the interval between attempts grows with each failure) and a maximum retry limit, after which the message is routed to a DLQ instead of being retried indefinitely.

Throughput measures how many messages a producer can publish, or a consumer can process, per second. It's the capacity metric that, combined with consumer lag, diagnoses whether a pipeline is healthy: consumption throughput lower than production throughput, sustained over time, is exactly what makes consumer lag grow indefinitely. Increasing consumption throughput usually means increasing parallelism (more partitions and consumers, Chapter 6) or reducing the processing cost per message — rarely does it mean tuning throughput without understanding which side is the bottleneck.

A topic is the abstraction producers and consumers see — for example, pagamentos.aprovados. It has no schema imposed by Kafka itself; what it guarantees is the split into partitions, each an ordered, immutable log. Delivery order is guaranteed within each partition of the topic, not across different topics, and not necessarily across partitions of the same topic.