Containerized Kafka client errors when producing messages to the host Kafka server

Question

There are a number of similar types of queries on stackoverflow, but none quite match the problem that I am seeing.

I have a zookeeper/kafka setup on my server which

Answer 1

Aha! After further reading and the answers given above the solution came. As is often the case it is an easy one. A simple edit of the kafka server.properties file:

advertised.listeners=PLAINTEXT://192.168.2.80:9092

Also note, the parameter 'listeners' is not set in this file.

Answer 2

The problem is (slightly simplified) caused by how Kafka's protocol works. Given a list of "bootstrap servers" (e.g. localhost:9092), a Kafka client will contact those bootstrap servers, but then use the hostnames of the actual Kafka brokers as returned by the bootstrap servers (the broker's advertised.listeners config, depending on your Kafka/Docker setup, might be set to e.g. kafka:9092). So here, the client would talk to localhost:9092 for bootstrapping (which will work), but then switch to kafka:9092 (which will not work, "thanks" to the networking setup).

Fortunately there is a way to configure Kafka + Docker in a way that "just works", and it doesn't require shenanigans such as fiddling with your host's /etc/hosts file and such. As part of this you need to set a few (new) Kafka settings though, which were added in kafka's KIP-103: Separation of Internal and External traffic.

Here's a snippet for Docker Compose (docker-compose.yml) that demonstrates how to do this:

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:3.2.1
    hostname: zookeeper
    ports:
      - '32181:32181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181

  kafka:
    image: confluentinc/cp-kafka:3.2.1
    hostname: kafka
    ports:
      - '9092:9092'
      - '29092:29092'
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      # Following line is needed for Kafka versions 0.11+
      # in case you run less than 3 Kafka brokers in your
      # cluster because the broker config
      # `offsets.topic.replication.factor` (default: 3)
      # is now enforced upon topic creation
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Here, the key settings are:

• listener.security.protocol.map (which is being set via KAFKA_LISTENER_SECURITY_PROTOCOL_MAP)
• inter.broker.listener.name
• advertised.listeners

In the setup above, the containerized Kafka broker listens on localhost:9092 for access from your host machine (e.g. your Mac laptop) and on kafka:29092 for access from other containers.

A full end-to-end example is available at: https://github.com/confluentinc/cp-docker-images/blob/v3.2.1/examples/kafka-streams-examples/docker-compose.yml (documentation at http://docs.confluent.io/3.2.1/cp-docker-images/docs/tutorials/kafka-streams-examples.html).

Answer 3

Your producer (in the container) can't resolve the host name of your Linux guest OS which is returned in the Kafka producers initial metadata request to the bootstrap server. You can add it manually to the /etc/hosts file inside the container or add "--add-host" parameter to the docker run command that launches the image running your producer