I have written a code to calculate euclidean distance between columns in UDF format:
distance_udf = F.udf(lambda x,y: float(distance.euclidean(x, y)),FloatType())