Spark iteration logic to write the dataset filtered by date to parquet format failing OOM

Question

I have a scenario where i have dataset with date column and later i use the dataset in iteration to save the dataset into multiple partition files in parquet format. I do it