How to manage physical data placement of a dataframe across the cluster with pyspark?

Question

Say I have a pyspark dataframe \'data\' as follows. I want to partition the data by \"Period\". Rather I want each period of data to be stored on it\'s own partition (see

Answer 1

The approach should be to repartition first, to have the right number of partitions(number of unique periods), and then partition by the Period column before saving it.

from pyspark.sql import functions as F
n = data.select(F.col('Period')).distinct().count()

data.repartition(n)\
     .write \
     .partitionBy("Period")\
     .mode("overwrite")\
     .format("parquet")\
     .saveAsTable("testing")