Kafka in Kubernetes Cluster- How to publish/consume messages from outside of Kubernetes Cluster

Question

1. I have Kafka deployed and running in Kubernetes cluster. I am using this image from docker hub - https://hub.docker.com/r/cloudtrackinc/kubernetes-kafka/
2. I have 3

Answer 1

I solved this problem by using Confluent's Kafka REST proxy image.

https://hub.docker.com/r/confluentinc/cp-kafka-rest/

Documentation of the REST Proxy is here:

http://docs.confluent.io/3.1.2/kafka-rest/docs/index.html

Step A: Build a Kafka broker docker image using latest Kafka version

I used a custom built Kafka broker image based on the same image you used. You basically just need to update cloudtrackinc's image to use Kafka version 0.10.1.0 or otherwise it won't work. Just update the Dockerfile from cloudertrackinc's image to use the latest wurstmeister kafka image and rebuild the docker image.

- FROM wurstmeister/kafka:0.10.1.0

I set the ADVERTISED_HOST_NAME for each Kafka broker to POD's IP so each broker gets an unique URL.

- name: ADVERTISED_HOST_NAME
  valueFrom:
    fieldRef:
      fieldPath: status.podIP

Step B: Setup cp-kafka-rest proxy to use your Kafka broker cluster

Kafka Rest Proxy must be running within the same cluster as your Kafka broker cluster.

You need to provide two environment variables to the cp-kafka-rest image at the minimum for it to run. KAFKA_REST_HOST_NAME and KAFKA_REST_ZOOKEEPER_CONNECT. You can set KAFKA_REST_HOST_NAME to use POD's IP.

- name: KAFKA_REST_HOST_NAME
  valueFrom:
    fieldRef:
      fieldPath: status.podIP
- name: KAFKA_REST_ZOOKEEPER_CONNECT
  value: "zookeeper-svc-1:2181,zookeeper-svc-2:2181,zookeeper-svc-3:2181"

Step C: Expose the Kafka REST proxy as a service

spec: type: NodePort or LoadBalancer ports: - name: kafka-rest-port port: 8082 protocol: TCP

You can use NodePort or LoadBalancer to utilize single or multiple Kafka REST Proxy pods.

Pros and Cons of using Kafka REST proxy

Pros:

1. You can scale Kafka broker cluster easily
2. You do not have to expose Kakfa brokers outside of the cluster
3. You can use the loadbalancer with the Proxy.
4. You can use any type of client to access the Kafka cluster (i.e. curl). Very light weight.

Cons:

1. Another component/layer on top of the Kakfa cluster.
2. Consumers are created within the proxy pod. This will need to be keep track of by your REST client.
3. Performance is not ideal: REST instead of native Kafka protocol. Although if you deploy multiple proxies this might help a little bit. I would not use this setup for high volume traffic. For low volume message traffic this might be okay.

So if you can live with the issues above, then give Kafka Rest Proxy a try.