How to avoid reloading of same data into hdfs at the time of any failure in pyspark

Question

I have a pyspark program to perform business mapping and load the data into two hive external tables partitioned based on month end date.

Requested Scenario:<