Spark SQL and MySQL- SaveMode.Overwrite not inserting modified data

后端 未结 3 1419
一个人的身影
一个人的身影 2020-12-06 03:14

I have a test table in MySQL with id and name like below:

+----+-------+
| id | name  |
+----+-------+
| 1  | Name1 |
+----+-------+
| 2  | Name         


        
3条回答
  •  离开以前
    2020-12-06 03:34

    Reading and writing to same table.

    cols_df = df_2.columns
    
    broad_cast_var = spark_context.broadcast(df_2.collect())
    
    df_3 = sqlContext.createDataFrame(broad_cast_var.value, cols_df)
    

    Reading and writing to same table with some modification.

    cols_df = df_2.columns
    
    broad_cast_var = spark_context.broadcast(df_2.collect())
    
    
    def update_x(x):
        y = (x[0] + 311, *x[1:])
        return y
    
    
    rdd_2_1 = spark_context.parallelize(broad_cast_var.value).map(update_x)
    
    df_3 = sqlContext.createDataFrame(rdd_2_1, cols_df)
    

提交回复
热议问题