I\'d like to stop various messages that are coming on spark shell.
I tried to edit the log4j.properties file in order to stop these message.
Her
Right after starting spark-shell type ;
sc.setLogLevel("ERROR")
In Spark 2.0 (Scala):
spark = SparkSession.builder.getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
API Docs : https://spark.apache.org/docs/2.2.0/api/scala/index.html#org.apache.spark.sql.SparkSession
For Java:
spark = SparkSession.builder.getOrCreate();
spark.sparkContext().setLogLevel("ERROR");