How to show full column content in a Spark Dataframe?

Question

I am using spark-csv to load data into a DataFrame. I want to do a simple query and display the content:

val df = sqlContext.read.format(\"com.databricks.spa         


        

Answer 1

results.show(20, false) will not truncate. Check the source

Answer 2

try this command :

df.show(df.count())

Answer 3

Within Databricks you can visualize the dataframe in a tabular format. With the command:

display(results)

It will look like

Answer 4

The other solutions are good. If these are your goals:

1. No truncation of columns,
2. No loss of rows,
3. Fast and
4. Efficient

These two lines are useful ...

    df.persist
    df.show(df.count, false) // in Scala or 'False' in Python

By persisting, the 2 executor actions, count and show, are faster & more efficient when using persist or cache to maintain the interim underlying dataframe structure within the executors. See more about persist and cache.

Answer 5

The following answer applies to a Spark Streaming application.

By setting the "truncate" option to false, you can tell the output sink to display the full column.

val query = out.writeStream
          .outputMode(OutputMode.Update())
          .format("console")
          .option("truncate", false)
          .trigger(Trigger.ProcessingTime("5 seconds"))
          .start()

Answer 6

results.show(20,false) did the trick for me in Scala.