Convert a spark DataFrame to pandas DF

Question

Is there a way to convert a Spark Df (not RDD) to pandas DF

I tried the following:

var some_df = Seq(
 (\"A\", \"no\"),
 (\"B\", \"yes\"),
 (\"B\", \         


        

Answer 1

Converting spark data frame to pandas can take time if you have large data frame. So you can use something like below:

spark.conf.set("spark.sql.execution.arrow.enabled", "true")

pd_df = df_spark.toPandas()

I have tried this in DataBricks.