Count empty values in dataframe column in Spark (Scala)

后端 未结 2 1890
不知归路
不知归路 2021-01-16 09:07

I\'m trying to count empty values in column in DataFrame like this:

df.filter((df(colname) === null) || (df(colname) === \"\")).count()

In

2条回答
  •  半阙折子戏
    2021-01-16 09:28

    As mentioned on the question that df.filter((df(colname) === null) || (df(colname) === "")).count() works for String data types but the testing shows that null are not handled.

    @Psidom's answer handles both null and empty but does not handle for NaN.

    checking for .isNaN should handle all three cases

    df.filter(df(colName).isNull || df(colName) === "" || df(colName).isNaN).count()
    

提交回复
热议问题