Partitions and ordering

Chapter 3 defined a partition as "the actual physical log." This chapter explains why that specific piece exists, how it turns the choice of a key into an architectural decision, and why "Kafka guarantees ordering" is an incomplete sentence until you say within what.

Without partitions, adding more consumers to a Consumer Group wouldn't increase consumption throughput — there would only be a single sequential stream to process. With partitions, consumption parallelism scales along with the topic's partition count.

• Consumption parallelism: within a Consumer Group, each partition is assigned to exactly one consumer at a time. More partitions means more consumers processing at the same time — up to the limit of one partition per consumer (Chapter 6 dives into what happens when that limit is exceeded).
• Storage and write scalability: partitions of the same topic can live on different brokers. A topic with 12 partitions spread across 3 brokers has 3 times more simultaneous write capacity than if it lived entirely on a single broker.

When a message does not have a key, the partitioner distributes messages across partitions in an approximately uniform way (recent versions of the Java client use a sticky strategy, grouping batches of messages into one partition at a time before rotating, optimizing batching without compromising distribution). The price of that distribution is that there's no ordering guarantee at all between those messages.

Ordering within a partition — and the absence of global ordering

This follows directly from the log structure: a partition is a linear, immutable sequence, so write order is, by construction, read order. But a topic with multiple partitions isn't a single sequence — it's a set of independent sequences, each with its own offset clock. Two messages in different partitions have no defined ordering relationship between them, even if one was visibly written before the other in real time.

• If two events for the same accountId need to be processed in the order they occurred (e.g., "PIX received" followed by "balance debited due to chargeback"), the key should be accountId — not eventId, which would give better distribution but would destroy the ordering between those two events.
• If the events are independent of each other (there's no ordering relationship that matters), optimizing distribution with a high-cardinality key (or no key) is a valid choice and improves load balancing across partitions.

Reducing the number of partitions of a topic, on the other hand, isn't natively supported by Kafka — the only way is to recreate the topic.

Architectural impacts of the partition choice

• Too few partitions limit the Consumer Group's maximum consumption parallelism and concentrate write load on few brokers.
• Too many partitions increase the cluster's metadata overhead, rebalance time (Chapter 6), and the number of open file handles per broker — with no added benefit if there aren't enough consumers to take advantage of the extra parallelism.
• The number of partitions should be sized based on target throughput and the maximum number of consumers the Consumer Group intends to scale to, not an arbitrary "round" value.