问题
Spark 2.0.0-preview
We've got an app that uses a fairly big broadcast variable. We run this on a big EC2 instance, so deployment is in client-mode. Broadcasted variable is a massive Map[String, Array[String]].
At the end of saveAsTextFile, the output in the folder seems to be complete and correct (apart from .crc files still being there) BUT the spark-submit process is stuck on, seemingly, removing the broadcast variable. The stuck logs look like this: http://pastebin.com/wpTqvArY
My last run lasted for 12 hours after after doing saveAsTextFile - just sitting there. I did a jstack on driver process, most threads are parked: http://pastebin.com/E29JKVT7
Full story:
We used this code with Spark 1.5.0 and it worked, but then the data changed and something stopped fitting into Kryo's serialisation buffer. Increasing it didn't help, so I had to disable the KryoSerialiser. Tested it again - it hanged. Switched to 2.0.0-preview - seems like the same issue.
I'm not quite sure what's even going on given that there's almost no CPU activity and no output in the logs, yet the output is not finalised like it used to before.
Would appreciate any help, thanks.
回答1:
I had a very similar issue.
I was updating from spark 1.6.1 to 2.0.1 and my steps were hanging after completion.
In the end, I managed to solve it by adding a sparkContext.stop() at the end of the task.
Not sure why this is needed it but it solved my issue. Hope this helps.
ps: this post reminds me of this https://xkcd.com/979/
来源：https://stackoverflow.com/questions/38311045/spark-stuck-at-removing-broadcast-variable-probably