How do I log from my Python Spark script

Question

I have a Python Spark program which I run with spark-submit. I want to put logging statements in it.

logging.info(\"This is an informative messa         


        

Answer 1

You can get the logger from the SparkContext object:

log4jLogger = sc._jvm.org.apache.log4j
LOGGER = log4jLogger.LogManager.getLogger(__name__)
LOGGER.info("pyspark script logger initialized")