Can pyspark.sql.function be used in udf?

Question

I define a function like

getDate = udf(lambda x : to_date(x))

When I use it in

df.select(getDate(\"time\")).show()
         


        

Answer 1

Looking at error seems problem with sc as sc._jvm:'NoneType' object has no attribute '_jvm'

Here sc is of NoneType.

And there is no need to write udf for it, you can use directly:-

import pyspark.sql.functions as F
df.select(F.to_date(df.time)).show()

Answer 2

Functions from pyspark.sql.functions are wrappers for JVM functions and are designed to operates on pyspark.sql.Column. You cannot use these:

• To transform local Python objects. They take Column and return Column.
• They cannot be used on the worker because there is no context in which they can be evaluated.