Databricks - Creating permanent User Defined Functions (UDFs)

Question

I am able to create a UDF function and register to spark using spark.UDF method. However, this is per session only. How to register python UDF functions automatically when the Cluster starts?. These functions should be available to all users. Example use case is to convert time from UTC to local time zone.

Answer 1

This is not possible; this is not like UDFs in Hive.

Code the UDF as part of the package / program you submit or in the jar included in the Spark App, if using spark-submit.

However,

spark.udf.register.udf("...

is required to be done as well. This applies to Databrick notebooks, etc. The UDFs need to be re-registered per Spark Context/Session.

Answer 2

acutally you can create a permanent function but not from a notebook you need to create it from a JAR file

https://docs.databricks.com/spark/latest/spark-sql/language-manual/create-function.html

CREATE [TEMPORARY] FUNCTION [db_name.]function_name AS class_name [USING resource, ...]

resource: : (JAR|FILE|ARCHIVE) file_uri