I am unable to send broadcast variable of size more than 1MB to udf. Here is the code
from pyspark.sql.functions import udf, col import numpy as np spark = S