Spark Exception : Task failed while writing rows

Question

I am reading text files and converting them to parquet files. I am doing it using spark code. But when i try to run the code I get following exception

org.ap         


        

Answer 1

In my case, I saw this error when I tried to overwrite hdfs directory which belonged to a different user. Deleting the directory a letting my process write it from scratch solved it. So I guess, more digging is appropriate in direction of user permissions on hdfs.