问题
I am trying to make use of mirror maker to replicate __consumre_offsets topic along with other topics.
It is giving error as mentioned below.
[2018-10-24 16:16:03,802] ERROR Error when sending message to topic __consumer_offsets with key: 16 bytes, value: 445 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic. [2018-10-24 16:16:03,828] ERROR Error when sending message to topic __consumer_offsets with key: 29 bytes, value: 754 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic.
Is there any way to resolve this?
In one of the confluent presentation in SlideShare on slide21, it mentioned about replicating offset topic in multiple datacenters. Can someone tell me the possible way to achieve the same?
Or there is any other better way for backup and restore policy for Kafka.
回答1:
Add this in your consumer.config:
exclude.internal.topics=false
And add this in your producer.config:
client.id=__admin_client
Reason being, Producers cannot write to internal topics unless you declare client.id=__admin_client which is used by AdminClient.scala. Found here: https://issues.apache.org/jira/browse/KAFKA-6524
回答2:
@amdelamar @Abhisek Verma Is this working for u. As suggested above i tried the same settings in my consumer and producer config files respectively and it still fails with the same error. It starts to replicate the topics and after a certain amount of time it keeps failing.
consumer.config
bootstrap.servers=https://kafka:9093
**exclude.internal.topics=false**
client.id=mirror_maker_consumer1
group.id=mirror_maker_consumer1
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="****" password="********";
ssl.protocol=TLSv1.2
ssl.enabled.protocols=TLSv1.2
ssl.endpoint.identification.algorithm=HTTPS
producer.config
bootstrap.servers=https://kafka-*****:9093
acks=1
**client.id=__admin_client**
batch.size=100
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="**************" password="****";
ssl.protocol=TLSv1.2
ssl.enabled.protocols=TLSv1.2
ssl.endpoint.identification.algorithm=HTTPS
ERROR Error when sending message to topic __consumer_offsets with key: 62 bytes, value: 28 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
来源：https://stackoverflow.com/questions/52967654/kafka-mirror-maker-failing-to-replicate-consumer-offset-topic