If you have less consumers than partitions, what happens?

Question

If you have less consumers than partitions, does that simply mean you will not consume all the messages on a given topic?

In a cloud environment, how are you suppose to

Answer 1

In fact, each consumer belongs to a consumer group. When Kafka cluster sends data to a consumer group, all records of a partition will be sent to a single consumer in the group.

If there're more paritions than consumers in a group, some consumers will consume data from more than one partition. If there're more consumers in a group than paritions, some consumers will get no data. If you add new consumer instances to the group, they will take over some partitons from old members. If you remove a consumer from the group (or the consumer dies), its partition will be reassigned to other member.

Now let's take a look at your questions:

If you have less consumers than partitions, does that simply mean you will not consume all the messages on a given topic?

NO. Some consumers in the same consumer group will consume data from more than one partition.

In a cloud environment, how are you suppose to keep track how many consumers are running and how many are pointing to a given topic#partition?

Kafka will take care of it. If new consumers join the group, or old consumers dies, Kafka will do reblance.

What if you have multiple consumers on a given topic#partition?

You CANNOT have multiple consumers (in a consumer group) to consume data from a single parition. However, if there're more than one consumer group, the same partition can be consumed by one (and only one) consumer in each consumer group.