How to create an dataframe from a dictionary where each item is a column in PySpark

问题

I want to make a new dataframe from a dictionary. The dictionary contains column names as keys and lists of columnar data as values. For example:

col_dict = {'col1': [1, 2, 3],
            'col2': [4, 5, 6]}

I need this as a dataframe that looks like this:

+------+------+
| col1 | col2 |
+------+------+
|     1|     4|
|     2|     5|
|     3|     6|
+------+------+

It doesn't seem like there's an easy way to do this.

回答1:

Easiest way is to create a pandas DataFrame and convert to a Spark DataFrame:

With Pandas

col_dict = {'col1': [1, 2, 3],
            'col2': [4, 5, 6]}

import pandas as pd
pandas_df = pd.DataFrame(col_dict)
df = sqlCtx.createDataFrame(pandas_df)
df.show()
#+----+----+
#|col1|col2|
#+----+----+
#|   1|   4|
#|   2|   5|
#|   3|   6|
#+----+----+

Without Pandas

If pandas is not available, you'll just have to manipulate your data into a form that works for the createDataFrame() function. Quoting myself from a previous answer:

I find it's useful to think of the argument to createDataFrame() as a list of tuples where each entry in the list corresponds to a row in the DataFrame and each element of the tuple corresponds to a column.

colnames, data = zip(*col_dict.items())
print(colnames)
#('col2', 'col1')
print(data)
#([4, 5, 6], [1, 2, 3])

Now we need to modify data so that it's a list of tuples, where each element contains the data for the corresponding column. Luckily, this is easy using zip:

data = zip(*data)
print(data)
#[(4, 1), (5, 2), (6, 3)]

Now call createDataFrame():

df = sqlCtx.createDataFrame(data, colnames)
df.show()
#+----+----+
#|col2|col1|
#+----+----+
#|   4|   1|
#|   5|   2|
#|   6|   3|
#+----+----+

来源：https://stackoverflow.com/questions/48796486/how-to-create-an-dataframe-from-a-dictionary-where-each-item-is-a-column-in-pysp