How to show full column content in a Spark Dataframe?

Question

I am using spark-csv to load data into a DataFrame. I want to do a simple query and display the content:

val df = sqlContext.read.format(\"com.databricks.spa         


        

Answer 1

In c# Option("truncate", false) does not truncate data in the output.

StreamingQuery query = spark
                    .Sql("SELECT * FROM Messages")
                    .WriteStream()
                    .OutputMode("append")
                    .Format("console")
                    .Option("truncate", false)
                    .Start();

Answer 2

Below code would help to view all rows without truncation in each column

df.show(df.count(), False)

Answer 3

results.show(false) will show you the full column content.

Show method by default limit to 20, and adding a number before false will show more rows.

Answer 4

Tried this in pyspark

df.show(truncate=0)

Answer 5

If you put results.show(false) , results will not be truncated

Answer 6

results.show(20, False) or results.show(20, false) depending on whether you are running it on Java/Scala/Python