How to get distinct rows in dataframe using pyspark?

问题:

I understand this is just a very simple question and most likely have been answered somewhere, but as a beginner I still don't get it and am looking for your enlightenment, thank you in advance:

I have a interim dataframe:

+----------------------------+---+ |host                        |day| +----------------------------+---+ |in24.inetnebr.com           |1  | |uplherc.upl.com             |1  | |uplherc.upl.com             |1  | |uplherc.upl.com             |1  | |uplherc.upl.com             |1  | |ix-esc-ca2-07.ix.netcom.com |1  | |uplherc.upl.com             |1  | 

What I need is to remove all the redundant items in host column, in another word, I need to get the final distinct result like:

+----------------------------+---+ |host                        |day| +----------------------------+---+ |in24.inetnebr.com           |1  | |uplherc.upl.com             |1  | |ix-esc-ca2-07.ix.netcom.com |1  | |uplherc.upl.com             |1  | 

回答1:

If df is the name of your DataFrame, there are two ways to get unique rows:

df2 = df.distinct() 

or

df2 = df.drop_duplicates() 

文章来源: How to get distinct rows in dataframe using pyspark?