How to stop INFO messages displaying on spark console?

Question

I\'d like to stop various messages that are coming on spark shell.

I tried to edit the log4j.properties file in order to stop these message.

Her

Answer 1

I just add this line to all my pyspark scripts on top just below the import statements.

SparkSession.builder.getOrCreate().sparkContext.setLogLevel("ERROR")

example header of my pyspark scripts

from pyspark.sql import SparkSession, functions as fs
SparkSession.builder.getOrCreate().sparkContext.setLogLevel("ERROR")