Spark Exception : Task failed while writing rows

Question

I am reading text files and converting them to parquet files. I am doing it using spark code. But when i try to run the code I get following exception

org.ap         


        

Answer 1

Another possible reason is that you're hitting s3 request rate limits. If you look closely at your logs you may see something like this

AmazonS3Exception: Please reduce your request rate.

While the Spark UI will say

Task failed while writing rows

I doubt its the reason you're getting an issue, but its a possible reason if you're running a highly intensive job. So I included just for answer's completeness.