What happens to consumer offsets if a new partition(s) is added to a Kafka topic?

Question

What will happen to consumer offsets when a new partition is added? Does it stay the same?

Answer 1

Assuming you're talking about the high level consumer/consumer groups, consumer offsets stay the same because offsets are specific to partitions. So, the consumer will begin consuming from the new partition at the beginning or end or at time t depending on the configured start offset. For instance, if the consumer has already consumed up to offset 100000 in partition 1 and 120000 in partition 2, when you add a third partition the consumer will discover the new partition after some period through polling and only reset the offset for that new partition.

One important thing to keep in mind when altering the set of partitions in a topic is hot spotting. If you add a new partition to the topic, historical messages will still be spread across the old partitions, so assuming you're using basic round-robin/hash partitioning the data in the old partitions will remain significantly larger than the data in the new partitions until all partitions have grown large enough for old segments of the old partitions to be deleted.