Get current number of partitions of a DataFrame

后端 未结 5 1106
太阳男子
太阳男子 2020-12-07 18:32

Is there any way to get the current number of partitions of a DataFrame? I checked the DataFrame javadoc (spark 1.6) and didn\'t found a method for that, or am I just missed

5条回答
  •  醉酒成梦
    2020-12-07 19:05

    One more Interesting way to get number of partitions is 'using mapPartitions' transformation. Sample Code -

    val x = (1 to 10).toList
    val numberDF = x.toDF()
    numberDF.rdd.mapPartitions(x => Iterator[Int](1)).sum()
    

    Spark experts are welcome to comment on its performance.

提交回复
热议问题