pyspark : NameError: name 'spark' is not defined

Question

I am copying the pyspark.ml example from the official document website: http://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.Transformer

Answer 1

Answer by 率怀一 is good and will work for the first time. But the second time you try it, it will throw the following exception :

ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=pyspark-shell, master=local) created by __init__ at :10 

There are two ways to avoid it.

1) Using SparkContext.getOrCreate() instead of SparkContext():

from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
sc = SparkContext.getOrCreate()
spark = SparkSession(sc)

2) Using sc.stop() in the end, or before you start another SparkContext.