cant connect to kafka from external machine

问题

I´m starting with Apache Kafka and i´m facing problems when i try to conect from an external machine.

With this configuration bellow, all works fine if the application and the docker are running at the same machine.

but when i put the application in machine A and docker at machine B, the application cant connect.

My spring Kafka @Configuration have this line to @Bean consumerFactory and producerFactory (imagine my machine with docker ip = 10.10.10.10)

props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "10.10.10.10:9092");

And my docker file is this:

version: '2'
services:
zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    ports:
      - 2181:2181
kafka:
    image: wurstmeister/kafka:0.10.1.1
    environment:
        KAFKA_ADVERTISED_HOST_NAME: 0.0.0.0
        KAFKA_ADVERTISED_PORT: 9092
        KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
        KAFKA_CREATE_TOPICS: "topic-jhipster:1:1,PROCESS_ORDER:1:1, PROCESS_CHANNEL:1:1"
        JMX_PORT: 9999
        KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=9999"
    ports:
        - 9092:9092
        - 9999:9999
kafka-manager:
    image: sheepkiller/kafka-manager
    ports:
        - 9000:9000
    links:
        - zookeeper
    environment:
        ZK_HOSTS: zookeeper:2181

i get this error:

org.springframework.kafka.core.KafkaProducerException: Failed to send;
nested exception is org.apache.kafka.common.errors.TimeoutException: 
Expiring 1 record(s) for

Edit, add some information..

I think its any configuration about the zookeeper i´m missing .. because if i have only the zookeeper started at my machine A .. and the kafka in machine B.. that works.. i only don´t know how :(

回答1:

Try setting Listeners,

eg: listeners = PLAINTEXT://your.host.name:9092

Assuming you can telnet between the machines on kafka port.

回答2:

Set advertised.listeners to the hostname or ip of the docker container's host machine.

environment:
        KAFKA_ADVERTISED_LISTENERS: "10.10.10.10:9092"

What is happening is that the client connects to the bootstrap server, does a meta-data request to discover which Kafka broker in the cluster to connect to for a particular topic partition, and gets the advertised host name back as a response (in your case 0.0.0.0) which will only work if everything is on the same machine.

You need to advertise an ip or hostname that will work from the remote machine so not localhost, 127.0.0.1, or 0.0.0.0. Also not the private internal IP or hostname of the docker container. It has to be the external/public IP or hostname.

Also advertised.host.name and advertised.port are deprecated parameters in Kafka 0.10.x so even if you use them (they work for backward compatibility), you also need to set the advertised.host.name to be something that the producer can resolve and connect to. I suggest using a fully qualified host name or IP of the docker host machine (I.e. 10.10.10.10).

From http://kafka.apache.org/0101/documentation.html#brokerconfigs

DEPRECATED: only used when advertised.listeners or listeners are not set. Use advertised.listeners instead. Hostname to publish to ZooKeeper for clients to use. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, it will use the value for host.name if configured. Otherwise it will use the value returned from java.net.InetAddress.getCanonicalHostName().

回答3:

Some mix of @Krishas and @Hans Jespersen

Here is the code of my docker yml:

version: '2'
services:
    zookeeper:
        image: wurstmeister/zookeeper:3.4.6
        ports:
          - 2181:2181
    kafka:
        image: wurstmeister/kafka:0.10.1.1
        environment:
            KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://10.10.10.10:9092
            KAFKA_ADVERTISED_HOST_NAME: 10.10.10.10
            KAFKA_ADVERTISED_PORT: 9092
            KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

That needs the "PLAINTEXT:// prefix ! And config the "host_name" + "port", or "listeners"

The next step is decovery how i will configure another nodes