pyspark-sql

So I have a spark dataframe that looks like: a | b | c 5 | 2 | 1 5 | 4 | 3 2 | 4 | 2 2 | 3 | 7 And I want to group by column a , create a list of values from column b, and forget about c. The output dataframe would be : a | b_list 5 | (2,4) 2 | (4,3) How would I go about doing this with a pyspark sql dataframe? Thank you! :) Here are the steps to get that Dataframe. >>> from pyspark.sql import functions as F >>> >>> d = [{'a': 5, 'b': 2, 'c':1}, {'a': 5, 'b': 4, 'c':3}, {'a': 2, 'b': 4, 'c':2}, {'a': 2, 'b': 3,'c':7}] >>> df = spark.createDataFrame(d) >>> df.show() +---+---+---+ | a| b| c| +--

In pyspark 1.6.2, I can import col function by from pyspark.sql.functions import col but when I try to look it up in the Github source code I find no col function in functions.py file, how can python import a function that doesn't exist? It exists. It just isn't explicitly defined. Functions exported from pyspark.sql.functions are thin wrappers around JVM code and, with a few exceptions which require special treatment, are generated automatically using helper methods. If you carefully check the source you'll find col listed among other _functions . This dictionary is further iterated and