SPARK DataFrame: How to efficiently split dataframe for each group based on same column values

Question

I have a DataFrame generated as follows:

df.groupBy($\"Hour\", $\"Category\")
  .agg(sum($\"value\").alias(\"TotalValue\"))
  .sort($\"Hour\".asc,$\"TotalVal         


        

Answer 1

//If you want to divide a dataset into n number of equal datasetssets
double[] arraySplit = {1,1,1...,n}; //you can also divide into ratio if you change the numbers.

List> datasetList = dataset.randomSplitAsList(arraySplit,1);