PySpark - Pass list as parameter to UDF

Question

I need to pass a list into a UDF, the list will determine the score/category of the distance. For now, I am hard coding all distances to be the 4th score.

a=         


        

Answer 1

Try currying the function, so that the only argument in the DataFrame call is the name of the column on which you want the function to act:

udf_score=udf(lambda x: cate(label_list,x), StringType())
a.withColumn("category", udf_score("distances")).show(10)