Can pyspark.sql.function be used in udf?

Question

I define a function like

getDate = udf(lambda x : to_date(x))

When I use it in

df.select(getDate(\"time\")).show()
         


        

Answer 1

Looking at error seems problem with sc as sc._jvm:'NoneType' object has no attribute '_jvm'

Here sc is of NoneType.

And there is no need to write udf for it, you can use directly:-

import pyspark.sql.functions as F
df.select(F.to_date(df.time)).show()