How to append data to an existing parquet file

Question

I\'m using the following code to create ParquetWriter and to write records to it.

ParquetWriter parquetWriter = new ParquetWriter(path,          


        

Answer 1

There is a Spark API SaveMode called append: https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/SaveMode.html which I believe solves your problem.

Example of use:

df.write.mode('append').parquet('parquet_data_file')