How to install Kafka on Windows?

Question

I\'m trying to install Kafka message queue on Windows for testing purposes (not for production).

I found this article on how to install Apache Kafka

Answer 1

In version 0.9.0. there are scripts in "bin" folder for windows. You can use them.

Answer 2

Kafka & Zookeeper have inbuilt Windows specific scripts as verified in the latest version. As mentioned in other other answers, the directory for windows scripts will be under kafka_2.12-2.4.0\bin\windows for Kafka. Zookeeper scripts are under /bin only but ending with "*.cmd"

There will be few minor additional steps you will have to do with ZooKeeper & Kafka like

• Creating zoo.cfg for ZooKeeper
• Configuring windows specific data directory for ZeeKeeper
• Kafka comes with default configuration file & it has very good OS specific handling i.e. log.dirs=/tmp/kafka-logs but it will create that directory structure under C:/ directory seamlessly without failing.

It is also easy to quickly setup a Kafka Cluster with multiple Kafka Brokers on Windows with minor configurations. You can also create topics, publish/consume messages, verify topics/partitions etc. all from Windows Command Prompt. Here is a good detailed reference for the windows specific setup Kafka Setup on Windows OS | Basic Installation, Setup, Verification, Cluster Setup, Storage

Answer 3

Yes, I had configured ZOOKEEPER and Apache Kafka on my Windows machine. For ZOOKEEPER, simply configure ZOOKEEPER_HOME and set the path also and then rename the zoo.sample.cfg to zoo.cfg and also change the path for dataDir in the zoo.cfg

For Apache Kafka, simply unzip and start the server from bin folder and start the Producer/Consumer from windows folder under the Kafka directory

Answer 4

With Chocolatey and Powershell:

1. chocolatey.exe install kafka -> installs Kafka into C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ - your version might differ of course
2. add C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ to your powershell path using notepad $PROFILE

After restarting powershell kafka-console-consumer should work as a command.

You also need to install ssl.properties with keystore and truststore. Put them e.g. into C:\ProgramData\kafka and put that into ssl.properties (please note the escaping of the backslashes):

security.protocol=SSL

ssl.truststore.location=C:\\ProgramData\\kafka\\kafka-truststore.jks
ssl.truststore.password=PASSWORD

ssl.keystore.location=C:\\ProgramData\\kafka\\kafka-keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD

client.id=console-test

Now things like kafka-consumer-groups --bootstrap-server SERVERNAME:9094 --command-config C:\ProgramData\kafka\ssl.properties --list should first pop up a firewall warning which you'd need to accept and then output a list of groups.

Answer 5

Running From A windows Shell

As of April 2019, downloading Kafka from their website worked on Windows almost right out of the box

Downloading and using the windows version .bat files is described here: https://kafka.apache.org/quickstart

I ran into two problems when I did this:

1) JAVA_HOME was set to an unsupported JDK which led to this kind of error

Exception in thread "main" java.lang.VerifyError: Uninitialized object exists on backward branch 209

Replacing with JDK 11 solved the problem.

2) The JAVA_HOME must not contain spaces which caused a 'cannot find the specified file' error. To fix this I used a shortened path like set JAVA_HOME=C:\Progra~1\Java\jdk-11.0.1

---

Running From Cygwin

If instead of the .bat files, you want to run the .sh files from cygwin, there is actually quite a bit that needs to be done, and even after that there may be problems that come up later. I can't really recommend this option, but I do use it as it's pretty convenient for some purposes.

If your JAVA_HOME path contains a space e.g. "C:\Program Files\Java\Jdk..." you will see something like this:

bin/kafka-run-class.sh: line 305: exec: C:\Program: not found

One solution is to copy the jdk to a path without spaces, and change the Java home accordingly.

If you don't want to change the JDK location, you can change the cygwin env variable as follows:

JAVA_HOME="/cygdrive/c/Program Files/Java/jdk-11.0.1"

and change the line

exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@"

to

exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@"

There was another problem with the log parameters in kafka-run-class.sh and I had to replace the line

KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"

with

KAFKA_GC_LOG_OPTS="-Xlog:gc*"

And even after all these changes I occasionally run into problems of Kafka shutting down because of an incompatible windows style path in the logs directories as described here: Kafka 1.0 stops with FATAL SHUTDOWN error. Logs directory failed In short, you may be better off running the Kafka .bat scripts from the windows directory...

Answer 6

1. Download any binary version(I've downloaded kafka_2.11-0.10.2.0) of Kafka from https://kafka.apache.org/downloads
2. Extract it to any folder(I've extracted to "C:\Kafka")
3. Open Command Prompt
4. Go to the folder that you extract Kafka(C:\Kafka\kafka_2.11-0.10.2.0)
5. Run this command .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties to start Zookeeper
6. Run this command .\bin\windows\kafka-server-start.bat .\config\server.properties to start Kafka

Now it works!