Kafka partitions have leader brokers without a matching listener

问题

I'm trying to get Kafka to work on docker-compose for the first time. The application runs fine without docker. But on docker, I get the error as described below. Any reason why Kafka would throw this error?

The error:

email-service_1 | 2018-12-01 14:32:02.448 WARN 1 --- [ntainer#0-0-C-1] o.a.k.c.NetworkClient : [Consumer clientId=consumer-2, groupId=kafka] 1 partitions have leader brokers without a matching listener, including [email-token-0]

My docker-compose config:

version: '3.3'
    services:
     zookeeper:
      image: wurstmeister/zookeeper
      ports:
       - "2181:2181"

 kafka:
  image: wurstmeister/kafka
  command: [start-kafka.sh]
  environment:
   KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
   KAFKA_ADVERTISED_HOST_NAME: 192.168.23.134
   KAFKA_CREATE_TOPICS: "email-token:1:1"
  volumes:
   - /var/run/docker.sock:/var/run/docker.sock
  ports:
   - "9092:9092"
  depends_on:
   - zookeeper

 email-service:
  build: ./email-service
  environment:
   SPRING_KAFKA_BOOTSTRAPSERVERS: kafka:9092
  ports:
   - "8081:8081"
  depends_on:
   - kafka

回答1:

As stated in the comments to your question the problem seems to be with the advertised name for the Kafka broker. According to your docker-compose you should be using 192.168.23.134 but your email-service is using kafka:9092. You can try with this docker-compose. I replaced the wurstmeister services with the latest Zookeeper and Kafka provided by confluentinc and added your email-service.

---
version: '2'
services:
zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
    ZOOKEEPER_CLIENT_PORT: 2181
    ZOOKEEPER_TICK_TIME: 2000

kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
    - zookeeper
    ports:
    - 9092:9092
    environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
    KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

email-service:
  build: ./email-service
  environment:
   SPRING_KAFKA_BOOTSTRAPSERVERS: kafka:29092
  ports:
   - "8081:8081"
  depends_on:
   - kafka

advertised.listeners: Listeners to publish to ZooKeeper for clients to use, if different than the listeners config property. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, the value for listeners will be used. Unlike listeners it is not valid to advertise the 0.0.0.0 meta-address.

Please note that KAFKA_ADVERTISED_HOST_NAME has been deprecated and it's recommended to use KAFKA_ADVERTISED_LISTENERS instead. For more information about KAFKA_ADVERTISED_LISTENERS check here.

回答2:

[context]

I am trying to run a docker compose with a kafka client using the registry of https://github.com/wurstmeister/kafka-docker I am trying to run a very simple kafka cluster with a single broker and 3 topics with each 1 partition and a replication factor of 1.

this great link explains connectivity for a kafka cluster with one broker, a kafka cluster with several brokers and also notions about the listeners, all using docker, please have a look : https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

[result]

The first time i run docker-compose up --force-recreate --build, everything runs just fine !
The topics are created automatically using KAFKA_CREATE_TOPICS and I can use kafka producer and consumer just fine.

list topics : bin/kafka-topics.sh --list --bootstrap-server localhost:9092
producer : bin/kafka-console-producer.sh --broker-list localhost:9092 --topic productadvisor_sales_dev
consumer : bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic productadvisor_sales_dev --from-beginning

After that, everytime I do docker-compose stop, and relaunch using docker-compose up --force-recreate --build and try to produce data I get the following error message ...

Error Message : [2019-09-23 19:41:33,037] WARN [Producer clientId=console-producer] 1 partitions have leader brokers without a matching listener, including [productadvisor_purchase_dev-0] (org.apache.kafka.clients.NetworkClient)

[Answer]

It appears you need to specify the value of KAFKA_BROKER_ID (=1 for instance) so that the zookeeper doesn't try to create a new broker which can't have a listener because it is binded to the old one.

[Code]


services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_HOST: localhost
      KAFKA_PORT: 9092
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "productadvisor_sales_dev:1:1,productadvisor_stock_dev:1:1,productadvisor_purchase_dev:1:1"
    depends_on:
      - zookeeper
    command: [start-kafka.sh]

[Some documentation]

https://rmoff.net/2018/08/02/kafka-listeners-explained/
https://www.tutorialspoint.com/apache_kafka/apache_kafka_fundamentals.htm
http://www.michael-noll.com/blog/2013/03/13/running-a-multi-broker-apache-kafka-cluster-on-a-single-node/
https://blog.k2datascience.com/running-kafka-using-docker-332207aec73c

if anyone has more information about the inner working of kafka, the zookeeper and the broker and why we need to specify it, why the information is kept even if I do a --force-recreate --build ... please do not hesitate. I am new to kafka and this is one of my first complete post on stackoverflow :)

Cheers !

来源：https://stackoverflow.com/questions/53571823/kafka-partitions-have-leader-brokers-without-a-matching-listener

标签

spring-boot

Docker

apache-kafka

docker-compose