I need to use the
(rdd.)partitionBy(npartitions, custom_partitioner)
method that is not available on the DataFrame. All of the DataFrame
Answer given by kennyut/Kistian works very well but to get exact RDD like output when RDD consist of list of attributes e.g. [1,2,3,4] we can use flatmap command as below,
rdd = df.rdd.flatMap(list) or rdd = df.rdd.flatmap(lambda x: list(x))