Question 16 of 50
How do you choose the number of Partitions?
Question
"How would you decide how many Partitions a topic should have?"
What the interviewer wants to assess
Sizing judgment — not a generic "it depends" answer, but concrete criteria and the consequences of under- or over-provisioning.
Resposta rápida
Two main variables: the topic's target throughput divided by a single partition's sustainable throughput, and the maximum number of consumers the Consumer Group plans to scale to in the future — since the number of partitions is the real ceiling on consumption parallelism.
Resposta nível Sênior
I start from expected throughput: if a partition sustains, say, a few MB/s of writes and the topic needs throughput well above that, I need enough partitions to split that load across brokers. Next, I consider the desired consumption parallelism — if the team plans to scale the consumer up to 20 replicas at peak, the topic needs at least 20 partitions, or 15 of them will sit idle (Chapter 6). I'd rather overshoot a bit than undershoot, because increasing partitions later changes the partitioning hash's result for new messages (Chapter 4), breaking the assumption that an entire key's history lives in a single partition — in general I'd rather migrate to a new topic than increase partitions on an already-in-production topic with meaningful history.
In-depth explanation
See "Increasing the number of partitions: what changes" and "Architectural impacts" in Chapter 4, and the parallelism ceiling in Chapter 6.
Exemplo financeiro
A card transactions topic that currently processes 500 transactions/second, with a plan to grow to 5,000
in two years, is sized with that ceiling already in mind — avoiding a mid-way partition increase, which
would break historical ordering by cartaoId.
"I always use 12 partitions, it is a nice round number"
A fixed number with no relation to target throughput or planned consumption parallelism is an arbitrary choice, not a sizing decision. It might happen to be right by accident, but it's not a defensible answer in an interview or in production.
Pode vir a seguir
Likely follow-ups: "what happens if you get that estimate wrong?" and "how does Kafka choose a Partition for each message?".
Related chapters