How to suppress parquet log messages in Spark?

Question

How to stop such messages from coming on my spark-shell console.

5 May, 2015 5:14:30 PM INFO: parquet.hadoop.InternalParquetRecordReader: at row 0. reading n         


        

Answer 1

I know this question was WRT Spark, but I recently had this issue when using Parquet with Hive in CDH 5.x and found a work-around. Details are here: https://issues.apache.org/jira/browse/SPARK-4412?focusedCommentId=16118403

Contents of my comment from that JIRA ticket below:

This is also an issue in the version of parquet distributed in CDH 5.x. In this case, I am using parquet-1.5.0-cdh5.8.4 (sources available here: http://archive.cloudera.com/cdh5/cdh/5)

However, I've found a work-around for mapreduce jobs submitted via Hive. I'm sure this can be adapted for use with Spark as well.

• Add the following properties to your job's configuration (in my case, I added them to hive-site.xml since adding them to mapred-site.xml didn't work:

  mapreduce.map.java.opts
  -Djava.util.logging.config.file=parquet-logging.properties


  mapreduce.reduce.java.opts
  -Djava.util.logging.config.file=parquet-logging.properties


  mapreduce.child.java.opts
  -Djava.util.logging.config.file=parquet-logging.properties

• Create a file named parquet-logging.properties with the following contents:

# Note: I'm certain not every line here is necessary. I just added them to cover all possible
# class/facility names.you will want to tailor this as per your needs.
.level=WARNING
java.util.logging.ConsoleHandler.level=WARNING

parquet.handlers=java.util.logging.ConsoleHandler
parquet.hadoop.handlers=java.util.logging.ConsoleHandler
org.apache.parquet.handlers=java.util.logging.ConsoleHandler
org.apache.parquet.hadoop.handlers=java.util.logging.ConsoleHandler

parquet.level=WARNING
parquet.hadoop.level=WARNING
org.apache.parquet.level=WARNING
org.apache.parquet.hadoop.level=WARNING

• Add the file to the job. In Hive, this is most easily done like so:

  ADD FILE /path/to/parquet-logging.properties;

With this done, when you run your Hive queries, parquet should only log WARNING (and higher) level messages to the stdout container logs.