How do I set the Java options for Kafka?

Question

I\'ve been experimenting with Kafka and saw from the documentation on the main site that you\'re able to set different options for the jvm like heap size and the garbage col

Answer 1

Modifying a script in the bin directory is highly unrecommended. When upgrading Kafka to the next version, extracting the new binaries would override the changes made in the script.

The preferred way should be to set the environment variable KAFKA_HEAP_OPTS outside the script.

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

If the var is set before starting Kafka via the script it will use the var instead of the default values defined in /bin/kafka-server-start.sh

Answer 2

You can also set this parameters via your service definition as shown below.

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
Environment="KAFKA_HEAP_OPTS=-Xmx4G"
Environment="KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.net.preferIPv4Stack=true"
Environment="JMX_PORT=9999"

[Install]
WantedBy=multi-user.target

Answer 3

Looking at kafka-run-classh.sh - kafka uses the following variables:

• $KAFKA_HEAP_OPTS
• $KAFKA_JVM_PERFORMANCE_OPTS
• $KAFKA_GC_LOG_OPTS
• $KAFKA_JMX_OPTS
• $KAFKA_LOG4J_OPTS

You can run it via:

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" 
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12346 -Dcom.sun.management.jmxremote.rmi.port=12346 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" 
bin/kafka-server-start.sh -daemon config/server.properties

Answer 4

Another way to do this is by modifying information written in /bin/kafka-server-start.sh:

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

or in /bin/kafka-run-class.sh:

KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"

Answer 5

To add generic JVM settings (for example user timezone) you could use KAFKA_OPTS environment variable from kafka-run-class.sh:

# Generic jvm settings you want to add
if [ -z "$KAFKA_OPTS" ]; then
  KAFKA_OPTS=""
fi

Answer 6

You can pass java parameters from command line . E.g.

java -server -Xms3072m -Xmx3072m -XX:NewSize=256m -XX:MaxNewSize=256m  -classpath <long list of jars> foo.class

For a `production server config you can create a properties file or set them in Code by creating

 Properties props = new Properties();
 props.put("serializer.class", "kafka.serializer.StringEncoder");

And then supply them to producerConfig

ProducerConfig config = new ProducerConfig(props);