Spark Exception : Task failed while writing rows

Question

I am reading text files and converting them to parquet files. I am doing it using spark code. But when i try to run the code I get following exception

org.ap         


        

Answer 1

I found that disabling speculation prevents this error from happening. I'm not very sure why. It seems that speculative and non-speculative tasks are conflicting when writing parquet rows.

sparkConf.set("spark.speculation","false")