update a dataframe column with new values

Question

df1 has fields id and json; df2 has fields idand json

df1.count() => 1200; df2.count()

Answer 1

If you want data from both the dataframe you can union two dataframe

import spark.implicits._

First Dataframe

val df1 = Seq(
  (1, "a"),
  (2, "b"),
  (3, "c")
).toDF("id", "value")    

Second dataframe

val df2 = Seq(
  (1, "x"), 
  (2, "y")
).toDF("id", "value")

To get the result as both the data from df1 and df2, use union

val resultDF = df1.union(df2)

resultDF.show()

Output :

+---+-----+
|id |value|
+---+-----+
|1  |a    |
|2  |b    |
|3  |c    |
|1  |x    |
|2  |y    |
+---+-----+