Interact with kafka docker container from outside of docker host [duplicate]

Answer 1

Here are my two cents, since I had a hard time figuring this one out.

My $KAFKA_HOME/config/server.properties contains the following:

listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT

advertised.listeners=INSIDE://${container_ip}:9092,OUTSIDE://${outside_host_ip}:29092

listeners=INSIDE://:9092,OUTSIDE://:29092

inter.broker.listener.name=INSIDE

This is creating two connections, one to be used inside docker and another to be used outside. You have to choose a new port for the latter, in my case 29092, make sure this port is exposed and mapped by docker.

I was not yet able to figure out a solution without the ${outside_host_ip} in the environment, hence I'm providing the host machine's ip as an env var.

Test:

1. Enter the Kafka container and create a topic: ./kafka-topics.sh -zookeeper zookeeper:2181 --create --topic dummytopic --partitions 1 --replication-factor 1
2. From outside the Kafka container do: ./kafka-console-producer.sh --broker-list 0.0.0.0:29092 --topic dummytopic and input a message

I hope this helps others

Answer 2

TL;DR Expose port 9092 on host and map it to 9092 container port to access kafka brokers outside container. See docker-compose documentation for details.

I think the problem is that you don't expose port 9092 outside container. According to your docker ps listing your 9092 container port is mapped dynamically to host's port range 32882-32884. When you connect to broker configured this way you receive metadata that contains port 9092 for advertising. With this metadata producer tries to do other requests through port 9092 and fails.

Answer 3

In the Kafka server properties you need to set advertised.host.name and advertised.portto the ip/port of your running container and then it should work.

Answer 4

You need to put the name of the host machine where the docker instance was deployed. Also you need to map ports from docker host machine(public) to the docker container instance(private).

Answer 5

For the record, another way to get my local kafka consumer communicating with remote broker inside a Docker container was to add an entry in my /etc/hosts : docker-host-ip-address docker-kafka-container-hostname

Anyway Lundahl's solution worked fine for me and seems cleaner. Even cleaner would be to set advertised.listeners=host-ip:port since advertised.host.name and advertised.port are deprecated.