How to stop INFO messages displaying on spark console?

Question

I\'d like to stop various messages that are coming on spark shell.

I tried to edit the log4j.properties file in order to stop these message.

Her

Answer 1

tl;dr

For Spark Context you may use:

sc.setLogLevel()

where loglevel can be ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE or WARN.

---

Details-

Internally, setLogLevel calls org.apache.log4j.Level.toLevel(logLevel) that it then uses to set using org.apache.log4j.LogManager.getRootLogger().setLevel(level).

You may directly set the logging levels to OFF using:

LogManager.getLogger("org").setLevel(Level.OFF)

You can set up the default logging for Spark shell in conf/log4j.properties. Use conf/log4j.properties.template as a starting point.

Setting Log Levels in Spark Applications

In standalone Spark applications or while in Spark Shell session, use the following:

import org.apache.log4j.{Level, Logger}

Logger.getLogger(classOf[RackResolver]).getLevel
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

Disabling logging(in log4j):

Use the following in conf/log4j.properties to disable logging completely:

log4j.logger.org=OFF

---

Reference: Mastering Spark by Jacek Laskowski.