What is the maximum size for a broadcast object in Spark?
问题 When using Dataframe broadcast function or the SparkContext broadcast functions, what is the maximum object size that can be dispatched to all executors? 回答1: broadcast function : Default is 10mb but we have used till 300 mb which is controlled by spark.sql.autoBroadcastJoinThreshold. AFAIK, It all depends on memory available. so there is no definite answer for this. what I would say is, it should be less than large dataframe and you can estimate large or small dataframe size like below...