I've heard conflicting advice about partition numbers. Some say more partitions mean better parallelism, while others warn about overhead. I'm trying to figure out how partition count actually affects throughput, consumer group scaling, and controller performance.
Throughput and Parallelism
Kafka's parallelism is largely bounded by the number of partitions. If you have 100 partitions, you can theoretically scale to 100 consumer instances in a single group. But does this linear scaling hold in practice?
Overhead Considerations
Each partition consumes memory for its log segments and requires metadata tracking on the broker. At what point does adding partitions start to degrade performance due to increased disk I/O contention and larger metadata caches?
Also, how does partition count influence rebalancing frequency when consumers join or leave?
