How to get a value from the Row object in Spark Dataframe?

Question

for

averageCount = (wordCountsDF
                .groupBy().mean()).head()

I get

Row(avg(count)=1.6666666666666667)

Answer 1

This also works:

averageCount = (wordCountsDF
                .groupBy().mean('count').collect())[0][0]
print averageCount

Answer 2

Dataframe rows are inherited from namedtuples (from the collections library), so while you can index them like a traditional tuple the way you did above, you probably want to access it by the name of its fields. That is, after all, the point of named tuples, and it is also more robust to future changes. Like this:

averageCount = wordCountsDF.groupBy().mean().head()['avg(jobs)']

Answer 3

I figured it out. This will return me the value:

averageCount = (wordCountsDF
                .groupBy().mean()).head()[0]