Pyspark: display a spark data frame in a table format

Question

I am using pyspark to read a parquet file like below:

my_df = sqlContext.read.parquet(\'hdfs://myPath/myDB.db/myTable/**\')

Then when I do

Answer 1

As mentioned by @Brent in the comment of @maxymoo's answer, you can try

df.limit(10).toPandas()

to get a prettier table in Jupyter. But this can take some time to run if you are not caching the spark dataframe. Also, .limit() will not keep the order of original spark dataframe.