Kubenetes: Is it possible to hit multiple pods with a single request in Kubernetes cluster

Question

I want to clear cache in all the pods in my Kubernetes namespace. I want to send one request to the end-point which will then send a HTTP call to all the pods in the namespace to clear cache. Currently, I can hit only one pod using Kubernetes and I do not have control over which pod would get hit.

Even though the load-balancer is set to RR, continuously hitting the pods(n number of times, where n is the total number of pods) doesn't help as some other requests can creep in.

The same issue was discussed here, but I couldn't find a solution for the implementation: https://github.com/kubernetes/kubernetes/issues/18755

I'm trying to implement the clearing cache part using Hazelcast, wherein I will store all the cache and Hazelcast automatically takes care of the cache update.

If there is an alternative approach for this problem, or a way to configure kubernetes to hit all end-points for some specific requests, sharing here would be a great help.

Answer 1

Provided you got kubectl in your pod and have access to the api-server, you can get all endpoint adressess and pass them to curl:

kubectl get endpoints <servicename> \
        -o jsonpath="{.subsets[*].addresses[*].ip}" | xargs curl

Alternative without kubectl in pod:

the recommended way to access the api server from a pod is by using kubectl proxy: https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod this would of course add at least the same overhead. alternatively you could directly call the REST api, you'd have to provide the token manually.

APISERVER=$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")
TOKEN=$(kubectl describe secret $(kubectl get secrets \
     | grep ^default | cut -f1 -d ' ') | grep -E '^token' | cut -f2 -d':' | tr -d " ")

if you provide the APISERVER and TOKEN variables, you don't need kubectl in your pod, this way you only need curl to access the api server and "jq" to parse the json output:

curl $APISERVER/api/v1/namespaces/default/endpoints --silent \
     --header "Authorization: Bearer $TOKEN" --insecure \
     | jq -rM ".items[].subsets[].addresses[].ip" | xargs curl

UPDATE (final version)

APISERVER usually can be set to kubernetes.default.svc and the token should be available at /var/run/secrets/kubernetes.io/serviceaccount/token in the pod, so no need to provide anything manually:

TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token); \
curl https://kubernetes.default.svc/api/v1/namespaces/default/endpoints --silent \
     --header "Authorization: Bearer $TOKEN" --insecure \
     | jq -rM ".items[].subsets[].addresses[].ip" | xargs curl

jq is available here: https://stedolan.github.io/jq/download/ (< 4 MiB, but worth it for easily parsing JSON)

Answer 2

For those of you trying to find an alternative, I have used hazelcast as distributed event listener. Added a similar POC on github: https://github.com/vinrar/HazelcastAsEventListener

Answer 3

I fixed this problem by using this script. You just have to write the equivalent command to make the API call. I used curl to do that.

Following is the usage of the script:

function usage {
    echo "usage: $PROGNAME [-n NAMESPACE] [-m MAX-PODS] -s SERVICE -- COMMAND"
    echo "  -s SERVICE   K8s service, i.e. a pod selector (required)"
    echo "     COMMAND   Command to execute on the pods"
    echo "  -n NAMESPACE K8s namespace (optional)"
    echo "  -m MAX-PODS  Max number of pods to run on (optional; default=all)"
    echo "  -q           Quiet mode"
    echo "  -d           Dry run (don't actually exec)"
}

For example to run command curl http://google.com on all pods of a service with name s1 and namespace n1, you need to execute ./kcdo -s s1 -n n1 -- curl http://google.com.