Spark SQL and MySQL- SaveMode.Overwrite not inserting modified data

Question

I have a test table in MySQL with id and name like below:

+----+-------+
| id | name  |
+----+-------+
| 1  | Name1 |
+----+-------+
| 2  | Name         


        

Answer 1

I believe all the steps above are unnecessary. Here's what you need to do:

• Create a dataset A like val A = spark.read.parquet("....")

• Read the table to be updated, as dataframe B. Make sure enable caching is enabled for dataframe B. val B = spark.read.jdbc("mytable").cache

• Force a count on B - this will force execution and cache the table depending on the chosen StorageLevel - B.count

• Now, you can do a transformation like val C = A.union(B)

• And, then write C back to the database like C.write.mode(SaveMode.Overwrite).jdbc("mytable")