AttributeError: 'NoneType' object has no attribute 'setCallSite'

Question

In PySpark, I want to calculate the correlation between two dataframe vectors, using the following code (I do not have any problem in importing pyspark or createDataFrame):

Answer 1

There are several reasons for getting that AttributeError:

1. You can use sc.stop before initializing one of xContext (where x could be SQL, Hive). For example:

   sc = SparkContext.getOrCreate(conf = conf)  
   sc.stop() 
   spark = SQLContext(sc)  
   

2. Your spark is not synchronized on a cluster.

So, just restart your jupyter notebook kernel or reboot an application (not spark context) and it will work.