How to run multiple kafka consumers on the same box independent of each other?

Question

I have two Kafka consumer ConsumerA and ConsumerB. I want to run these two kafka consumers independent of each other on the same machine. There is

Answer 1

A quick suggestion, apologies if you know about it already. Class level variables are never thread safe. If you need to have a different Properties object for every thread, better declare them at the method level and provide them as parameter to other methods where you need to access Properties object.

Answer 2

Easiest Solution to solve "What is the best way to solve this thread safety issue and still achieve the same features?" :

Do not implement multi threading (Thread API/Executor Service) but instead use and run each consumer as a single consumer in its own separate JVM process, so if you need 4 consumers on same machine and you dont want to deal with mutli threading headaches then have your kafka consumer code JAR run in its own 4 separate Java processes.

Answer 3

Try Apache Samza. It solves these consumer problems. No messy (and sometimes problematic) handling of threads, redundancy via clustering, proven solution by trillions of proven processed mesages, etc. We are currently running more than one job on the cluster. Our code is much less complex than what you have here.