Running PySpark on and IDE like Spyder?

Question

I could run PySpark from the terminal line and everything works fine.

~/spark-1.0.0-bin-hadoop1/bin$ ./pyspark

Welcome to

Answer 1

1. If you just want to import the module , adding it to python path is enough

2. If you want to run complete scripts from the IDE, you can create a 'tool' that uses spark-submit to execute your script from the IDE (instead of normal run)

3. Specifically for spyder (or other IDE's that are written in python) you can run the IDE from within spark-submit

example:

spark-submit.cmd c:\Python27\Scripts\spyder.py

• note that I had to rename spyder to spyder.py - it appears spark submit relies on the extension do distinguish between python, java, or scala
• add any required parameters to spark-submit