How do I log from my Python Spark script

Question

I have a Python Spark program which I run with spark-submit. I want to put logging statements in it.

logging.info(\"This is an informative messa         


        

Answer 1

import logging

# Logger

logging.basicConfig(format='%(asctime)s %(filename)s %(funcName)s %(lineno)d %(message)s')
logger = logging.getLogger('driver_logger')
logger.setLevel(logging.DEBUG)

Simplest way to log from pyspark !