key not found: _PYSPARK_DRIVER_CALLBACK_HOST

Question

I\'m trying to run this code:

import pyspark
from pyspark.sql import SparkSession

spark = SparkSession.builder \\
        .master(\"local\") \\
        .app         


        

Answer 1

pip install pyspark==2.3.0

is ok

Answer 2

I have got the similar errors: java.util.NoSuchElementException: key not found: _PYSPARK_DRIVER_CALLBACK_HOST and Exception: Java gateway process exited before sending its port number

Running the command "export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH" or setting this to .bashrc resolved the issue.

Please also check if the mapr credentails are setup.

Answer 3

This resolution that I'm about to render also takes care of the "key not found: _PYSPARK_DRIVER_CALLBACK_HOST/Java Gateway/PySpark 2.3.1" error!! Add to your bashrc or /etc/environment or /etc/profile

export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH

That should do the doobie right there. You may thank me in advance. #thumbsup :)

Answer 4

This error is a result of a version mismatch. Environment variable which is referenced in the traceback (_PYSPARK_DRIVER_CALLBACK_HOST) has been removed during update Py4j dependency to 0.10.7 and backported to 2.3 branch in 2.3.1.

Considering version information:

I'm using PyCharm and MacOS, Python 3.6, Spark 2.3.1

it looks like you have 2.3.1 package installed, but SPARK_HOME points to an older (2.3.0 or earlier) installation.

Answer 5

the env var in .bash_profile or /etc/profile may not be accessed by your code ,put them in your code directly.

import os
import sys

os.environ['SPARK_HOME'] = "/opt/cloudera/parcels/SPARK2/lib/spark2"
os.environ['PYSPARK_SUBMIT_ARGS'] = "--master yarn pyspark-shell" 




sys.path.append(os.path.join(os.environ['SPARK_HOME'], "python"))
sys.path.append(os.path.join(os.environ['SPARK_HOME'], "python/lib/py4j-0.10.6-src.zip"))


try:
    from pyspark import SparkContext
    from pyspark.sql import SparkSession

    from pyspark import SparkConf

    print("success")

except ImportError as e:
    print("error importing spark modules", e)
    sys.exit(1)

Answer 6

I had same issue and all above settings did not work for me. Actually i had SPARK_HOME already set. Finally the issue was because i simply installed pyspark using pip install pyspark without verifying the version. After a lot of debugging inside the code , figured out that the _PYSPARK_DRIVER_CALLBACK_HOST inside

anaconda3/lib/python3.7/site-packages/pyspark/java_gateway.py

did not have this variable whereas older versions of pyspark has it (I am using anaconda hence this file path location. The exact location of the file might be different for others)

Finally came to the conclusion that it was due to version mismatch. Seems very stupid enough , but i guess it might help others a lot of debugging time.

solution is to find out the spark version that is installed for eg 2.3.0 and then ensure to install pyspark of same version pip install pyspark==2.3.0 . After this it worked like a charm.

Note : this issue occurs only if i call SparkSession.builder.appName within python . It was working fine even with the version mismatch for pyspark and spark-submit commands , and that's why it easily skipped my mind that it can be due to version mismatch.