I am trying to write a very simple code using Spark in Pycharm and my os is Windows 8. I have been dealing with several problems which somehow managed to fix except for one.
After struggling with this for two days, I figured what the problem is. I added the followings to the "PATH" variable as windows environment variable:
C:/Spark/spark-1.4.1-bin-hadoop2.6/python/pyspark
C:\Python27
Remember, You need to change the directory to wherever your spark is installed and also the same thing for python. On the other hand, I have to mention that I am using prebuild version of spark which has Hadoop included.
Best of luck to you all.