How to suppress INFO messages for spark-sql running on EMR?

Question

I\'m running Spark on EMR as described in Run Spark and Spark SQL on Amazon Elastic MapReduce:

This tutorial walks you through installing and operating Sp

Answer 1

You can also just add the configuration option at cluster creation, if you know you want to suppress logging for a new EMR cluster.

EMR accepts configuration options as JSON, which you can enter directly into the AWS console, or pass in via a file when using the CLI.

In this case, in order to change the log level to WARN, here's the JSON:

[
  {
    "classification": "spark-log4j",
    "properties": {"log4j.rootCategory": "WARN, console"}
  }
]

In the console, you'd add this in the first creation step:

Or if you're creating the cluster using the CLI:

aws emr create-cluster  --configurations config_file.json

You can read more in the EMR documentation.