问题
I am wondering if there is a limit size of a broadcast variable in spark because I need to broadcast a large object (6GB and can be larger in the future). I have searched in the home page and this site but nothing was found to answer my question. Would I ever have problems with broadcasting a very large object?
回答1:
There is a limit of Java Integer.MAX_VALUE. That is to say about 2,3 GB. Please have a look here in Spark Jira for more information: https://issues.apache.org/jira/browse/SPARK-5739
来源：https://stackoverflow.com/questions/32021419/is-there-any-limit-on-size-of-a-spark-broadcast-variable