Why does spark-submit and spark-shell fail with “Failed to find Spark assembly JAR. You need to build Spark before running this program.”?

Question

I was trying to run spark-submit and I get \"Failed to find Spark assembly JAR. You need to build Spark before running this program.\" When I try to run spark-shell

Answer 1

In my case, I install spark by pip3 install pyspark on macOS system, and the error caused by incorrect SPARK_HOME variable. It works when I run command like below:

PYSPARK_PYTHON=python3 SPARK_HOME=/usr/local/lib/python3.7/site-packages/pyspark python3 wordcount.py a.txt