Automatically including jars to PySpark classpath

Question

I\'m trying to automatically include jars to my PySpark classpath. Right now I can type the following command and it works:

$ pyspark --jars /path/to/my.jar
         


        

Answer 1

Recommended way since Spark 2.0+ is to use spark.driver.extraLibraryPath and spark.executor.extraLibraryPath

https://spark.apache.org/docs/2.4.3/configuration.html#runtime-environment

ps. spark.driver.extraClassPath and spark.executor.extraClassPath are still there, but deprecated and will be removed in a future release of Spark.