问题
I have persisted machine learning model in hdfs via spark batch job and i am consuming this in my spark streaming. Basically, the ML model is broadcasted to all executors from the spark driver. Can some one suggest how i can update the model in real time without stopping the spark streaming job? Basically a new ML model will get created as and when more data points are available but not have any idea how the NEW model will need to be sent to the spark executors. Request to post some sample code as well.
Regards, Deepak.
回答1:
The best approach is probably updating the model on each batch. Since you would probably rather not update too often, you probably want to check if you actually need to load the model and skip that if possible.
In your case of a model stored on hdfs, you can just check for a new timestamp on the model file (or a new model present in a directory) before updating the value of the variable holding the loaded model.
来源：https://stackoverflow.com/questions/48749717/ml-model-update-in-spark-streaming