How to check which partition is a key assign to in kafka?

后端 未结 2 1448
滥情空心
滥情空心 2021-01-06 07:32

I am trying to debug a issue for which I am trying to prove that each distinct key only goes to 1 partition if the cluster is not rebalancing.

So I was wondering for

2条回答
  •  清歌不尽
    2021-01-06 08:27

    As explained here or also in the source code

    You need the byte[] keyBytes assuming it isn't null, then using org.apache.kafka.common.utils.Utils, you can run the following.

    Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
    

    For strings or JSON, it's UTF8 encoded, and the Utils class has helper functions to get that.
    For Avro, such as Confluent serialized values, it's a bit more complicated (a magic byte, then a schema ID, then the data). See Wire format

    only goes to 1 partition

    This isn't a guarantee. Hashes can collide.

    It makes more sense to say that a given key isn't in more than one partition.

    if the cluster is not rebalancing

    Rebalancing will still preserve a partition value.

提交回复
热议问题