PySpark converting a column of type 'map' to multiple columns in a dataframe

Question

Input

I have a column Parameters of type map of the form:

>>> from pyspark.sql import SQLContext
>>> s         


        

Answer 1

Since keys of the MapType are not a part of the schema you'll have to collect these first for example like this:

from pyspark.sql.functions import explode

keys = (df
    .select(explode("Parameters"))
    .select("key")
    .distinct()
    .rdd.flatMap(lambda x: x)
    .collect())

When you have this all what is left is simple select:

from pyspark.sql.functions import col

exprs = [col("Parameters").getItem(k).alias(k) for k in keys]
df.select(*exprs)