use an external library in pyspark job in a Spark cluster from google-dataproc

Question

I have a spark cluster I created via google dataproc. I want to be able to use the csv library from databricks (see https://github.com/databricks/spark-csv). So I f

Answer 1

Additionally to @Dennis.

Note that if you need to load multiple external packages, you need to specify a custom escape character like so:

--properties ^#^spark.jars.packages=org.elasticsearch:elasticsearch-spark_2.10:2.3.2,com.data‌​bricks:spark-avro_2.10:2.0.1

Note the ^#^ right before the package list. See gcloud topic escaping for more details.