key not found: _PYSPARK_DRIVER_CALLBACK_HOST

Question

I\'m trying to run this code:

import pyspark
from pyspark.sql import SparkSession

spark = SparkSession.builder \\
        .master(\"local\") \\
        .app         


        

Answer 1

This error is a result of a version mismatch. Environment variable which is referenced in the traceback (_PYSPARK_DRIVER_CALLBACK_HOST) has been removed during update Py4j dependency to 0.10.7 and backported to 2.3 branch in 2.3.1.

Considering version information:

I'm using PyCharm and MacOS, Python 3.6, Spark 2.3.1

it looks like you have 2.3.1 package installed, but SPARK_HOME points to an older (2.3.0 or earlier) installation.