kafka MirrorMaker : No broker partitions consumed by consumer thread kafka-mirror

问题

This is regarding kafka MirrorMaker tool.

I have configured kafka on two machines.

source:
destination: vm [ubuntu at the source only]

Kafka at both source and destination are of same version of kafka [kafka_2.11-0.9.0.0]

At source and destination, respective zookeeper and kafka servers are running.

with the MirrorMaker tool I wanted to replicate/make mirror of topics from source to destination.

Below is the command , that I have used:

./bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config ./config/mirror_consumer.properties --producer.config ./config/mirror_producer.properties --whitelist='.*' &>mirror-log.log

configuration files contains

a. mirror_consumer.properties

    #host:port of kafka source zookeeper to be mirrored
    zookeeper.connect=source-ip:3181
    zookeeper.connection.timeout.ms=1000000
    consumer.timeout.ms=-1
    security.protocol=PLAINTEXT
    group.id=kafka-mirror

    where, 

        source-ip is ip address of source machine.
        my zookeeper at source is running at port 3181.

b. mirror_producer.properties

    # mirror broker (local) at the destination
    bootstrap.servers=localhost:9092
    producer.type=async

   where,

       localhost, resolves to destination i.e. ubuntu vm
       and kafka is runnning on default port i.e. 9092

Initially, I have created few topics with name say source1 and source2. From source machine with respective producers from command line I have sent some messages to the topics created.

after executing the MirrorMaker command from destination,

I could see that the consumer at destination is trying to consume the topics. Unfortunately, consumer at destination fails to read the partitions from broker for each topic.

please have a look at the sample log entry below:

[2016-05-06 13:25:00,931] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462521159741-6c2475c3-0 for topic source1 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,931] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462521295337-c3742307-0 for topic source1 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,931] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462517840512-a134d048-0 for topic source2 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,932] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462519206297-63bc9c58-0 for topic source2 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,932] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462519513695-bee7950e-0 for topic source2 (kafka.consumer.RangeAssignor)

Please let me know , if you see anything that is missing / need to be fixed.

It would be great help.

Thanks in advance.

回答1:

We get this issue when there is a mismatch between the number of partitions in a topic to the number of consumers in a consumer group feeding to the same topic.

来源：https://stackoverflow.com/questions/37068720/kafka-mirrormaker-no-broker-partitions-consumed-by-consumer-thread-kafka-mirro

标签

apache-kafka

mirroring