pyspark : NameError: name 'spark' is not defined

Question

I am copying the pyspark.ml example from the official document website: http://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.Transformer

Answer 1

Since you are calling createDataFrame(), you need to do this:

df = sqlContext.createDataFrame(data, ["features"])

instead of this:

df = spark.createDataFrame(data, ["features"])

spark stands there as the sqlContext.

---

In general, some people have that as sc, so if that didn't work, you could try:

df = sc.createDataFrame(data, ["features"])