What is going wrong with `unionAll` of Spark `DataFrame`?

Question

Using Spark 1.5.0 and given the following code, I expect unionAll to union DataFrames based on their column name. In the code, I\'m using some FunSuite for pass

Answer 1

Use unionByName:

Excerpt from the documentation:

def unionByName(other: Dataset[T]): Dataset[T]

The difference between this function and union is that this function resolves columns by name (not by position):

val df1 = Seq((1, 2, 3)).toDF("col0", "col1", "col2")
val df2 = Seq((4, 5, 6)).toDF("col1", "col2", "col0")
df1.union(df2).show

// output:
// +----+----+----+
// |col0|col1|col2|
// +----+----+----+
// |   1|   2|   3|
// |   4|   5|   6|
// +----+----+----+