how to quieten output from spark-submit?

问题

I'm trying to automate spark-submit jobs for spark on bluemix, but I'm receiving a lot of output (e.g. file upload status). This verbose output is fine for manually running spark-submit, but it is just noise when trying to execute if from another script.

Is there a way to quieten the output from the bluemix custom spark-submit script?

---

Update:

The output I'm trying to quieten is that generated from the spark-submit script, for example:

Uploading ./truststore.jar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3103    0    99  100  3004     60   1835  0:00:01  0:00:01 --:--:--  1835

I am not trying to quieten the output from spark, so quietening log4j will not help.

回答1:

Not sure if this what exactly you want:-

Just redirect everything to /dev/null

./spark-submit.sh --vcap credentials/vcap.json --deploy-mode cluster --master https://x.x.x.x:8443 --files abc.txt hellopy.py file://abc.txt > /dev/null 2>&1

Thanks, Charles.

回答2:

you can quiet warnings and infos this way, if you use python, but shuld be similar in other languages too:

def quiet_logs(sc):
logger = sc._jvm.org.apache.log4j
logger.LogManager.getLogger("org").setLevel(logger.Level.ERROR)
logger.LogManager.getLogger("akka").setLevel(logger.Level.ERROR)

sc=SparkContext()
quiet_logs(sc)

来源：https://stackoverflow.com/questions/37219811/how-to-quieten-output-from-spark-submit