remove NULL columns in Spark SQL

Question

How to remove columns containing only null values from a table? Suppose I have a table -

SnapshotDate    CreationDate    Country Region  CloseDate   Probabi         


        

Answer 1

You can add custom udf, and it in Spark SQL.

sqlContext.udf.register("ISNOTNULL", (str: String) => Option(str).getOrElse(""))

And with Spark SQL you can do :

SELECT ISNOTNULL(Probability) Probability, ISNOTNULL(BookingAmount) BookingAmount, ISNOTNULL(RevenueAmount) RevenueAmount FROM df