How to run a script in PySpark

问题:

I'm trying to run a script in the pyspark environment but so far I haven't been able to. How can I run a script like python script.py but in pyspark? Thanks

回答1:

You can do: ./bin/spark-submit mypythonfile.py

Running python applications through pyspark is not supported as of Spark 2.0.

回答2:

pyspark 2.0 and later execute script file in environment variable PYTHONSTARTUP, so you can run:

PYTHONSTARTUP=code.py pyspark 

Compared to spark-submit answer this is useful for running initialization code before using the interactive pyspark shell.

回答3:

Just spark-submit mypythonfile.py should be enough.

文章来源: How to run a script in PySpark