Link Spark with iPython Notebook

Question

I have followed some tutorial online but they do not work with Spark 1.5.1 on OS X El Capitan (10.11)

Basically I have run this commands download

Answer 1

Spark with IPython/Jupyter notebook is great and I'm pleased the Alberto was able to help you get it working.

For reference it's also worth considering 2 good alternatives that come prepackaged and can easily be integrated into a YARN cluster (if desired.)

Spark Notebook: https://github.com/andypetrella/spark-notebook

Apache Zeppelin: https://zeppelin.incubator.apache.org/

At the time of writing Spark Notebook (v0.6.1) is more mature and you can prebuild an install against your Spark and Hadoop version here: http://spark-notebook.io/

Zeppelin (v0.5) looks very promising but doesn't offer as much functionality as Spark Notebook or IPython with Spark right now.