How to expose a headless service for a StatefulSet externally in Kubernetes

Question

Using kubernetes-kafka as a starting point with minikube.

This uses a StatefulSet and a headless service for service discovery within the cluster.

The goal i

Answer 1

Solutions so far weren't quite satisfying enough for myself, so I'm going to post an answer of my own. My goals:

1. Pods should still be dynamically managed through a StatefulSet as much as possible.
2. Create an external service per Pod (i.e Kafka Broker) for Producer/Consumer clients and avoid load balancing.
3. Create an internal headless service so that each Broker can communicate with each other.

Starting with Yolean/kubernetes-kafka, the only thing missing is exposing the service externally and two challenges in doing so.

1. Generating unique labels per Broker pod so that we can create an external service for each of the Broker pods.
2. Telling the Brokers to communicate to each other using the internal Service while configuring Kafka to tell the producer/consumers to communicate over the external Service.

Per pod labels and external services:

To generate labels per pod, this issue was really helpful. Using it as a guide, we add the following line to the 10broker-config.yml init.sh property with:

kubectl label pods ${HOSTNAME} kafka-set-component=${HOSTNAME}

We keep the existing headless service, but we also generate an external Service per pod using the label (I added them to 20dns.yml):

apiVersion: v1
kind: Service
metadata:
  name: broker-0
   namespace: kafka
spec:
  type: NodePort
  ports:
  - port: 9093
    nodePort: 30093
selector:
  kafka-set-component: kafka-0

Configure Kafka with internal/external listeners

I found this issue incredibly useful in trying to understand how to configure Kafka.

This again requires updating the init.sh and server.properties properties in 10broker-config.yml with the following:

Add the following to the server.properties to update the security protocols (currently using PLAINTEXT):

listener.security.protocol.map=INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT
inter.broker.listener.name=INTERNAL_PLAINTEXT

Dynamically determine the external IP and for external port for each Pod in the init.sh:

EXTERNAL_LISTENER_IP=
EXTERNAL_LISTENER_PORT=$((30093 + ${HOSTNAME##*-}))

Then configure listeners and advertised.listeners IPs for EXTERNAL_LISTENER and INTERNAL_LISTENER (also in the init.sh property):

sed -i "s/#listeners=PLAINTEXT:\/\/:9092/listeners=INTERNAL_PLAINTEXT:\/\/0.0.0.0:9092,EXTERNAL_PLAINTEXT:\/\/0.0.0.0:9093/" /etc/kafka/server.properties
sed -i "s/#advertised.listeners=PLAINTEXT:\/\/your.host.name:9092/advertised.listeners=INTERNAL_PLAINTEXT:\/\/$HOSTNAME.broker.kafka.svc.cluster.local:9092,EXTERNAL_PLAINTEXT:\/\/$EXTERNAL_LISTENER_IP:$EXTERNAL_LISTENER_PORT/" /etc/kafka/server.properties

Obviously, this is not a full solution for production (for example addressing security for the externally exposed brokers) and I'm still refining my understanding of how to also let internal producer/consumers to also communicate with the brokers.

However, so far this is the best approach for my understanding of Kubernetes and Kafka.