SPARK : failure: ``union'' expected but `(' found

Question

I have a dataframe called df with column named employee_id. I am doing:

 df.registerTempTable(\"d_f\")
val query = \"\"\"SELECT *, ROW_NUMBER() OVER (ORDER B         


        

Answer 1

Spark 2.0+

Spark 2.0 introduces native implementation of window functions (SPARK-8641) so HiveContext should be no longer required. Nevertheless similar errors, not related to window functions, can be still attributed to the differences between SQL parsers.

Spark <= 1.6

Window functions have been introduced in Spark 1.4.0 and require HiveContext to work. SQLContext won't work here.

Be sure you you use Spark >= 1.4.0 and create the HiveContext:

import org.apache.spark.sql.hive.HiveContext
val sqlContext = new HiveContext(sc)