How to suppress INFO messages for spark-sql running on EMR?

Question

I\'m running Spark on EMR as described in Run Spark and Spark SQL on Amazon Elastic MapReduce:

This tutorial walks you through installing and operating Sp

Answer 1

I was able to do this by editing $HOME/spark/conf/log4j.properties as desired, and calling spark-sql with --driver-java-options as follows:

./spark/bin/spark-sql --driver-java-options "-Dlog4j.configuration=file:///home/hadoop/spark/conf/log4j.properties"

I could verify that the correct file was being used by adding -Dlog4j.debug to the options:

./spark/bin/spark-sql --driver-java-options "-Dlog4j.debug -Dlog4j.configuration=file:///home/hadoop/spark/conf/log4j.properties"