How to load a model saved in joblib file from Google Cloud Storage bucket

问题

I want to load a model which is saved as a joblib file from Google Cloud Storage bucket. When it is in local path, we can load it as follows (considering model_file is the full path in system):

loaded_model = joblib.load(model_file)

How can we do the same task with Google Cloud Storage?

回答1:

I don't think that's possible, at least in a direct way. I though about a workaround, but the might not be as efficient as you want.

By using the Google Cloud Storage client libraries [1] you can download the model file first, load it, and when your program ends, delete it. Of course, this means that you need to download the file every time you run the code. Here is a snippet:

from google.cloud import storage
from sklearn.externals import joblib

storage_client = storage.Client()
bucket_name=<bucket name>
model_bucket='model.joblib'
model_local='local.joblib'

bucket = storage_client.get_bucket(bucket_name)
#select bucket file
blob = bucket.blob(model_bucket)
#download that file and name it 'local.joblib'
blob.download_to_filename(model_local)
#load that file from local file
job=joblib.load(model_local)

回答2:

For anyone googling around for an answer to this. Here are two more options besides the obvious, to use Google AI platform for model hosting (and online predictions).

Option 1 is to use TemporaryFile like this:

from google.cloud import storage
from sklearn.externals import joblib
from tempfile import TemporaryFile

storage_client = storage.Client()
bucket_name=<bucket name>
model_bucket='model.joblib'

bucket = storage_client.get_bucket(bucket_name)
#select bucket file
blob = bucket.blob(model_bucket)
with TemporaryFile() as temp_file:
    #download blob into temp file
    blob.download_to_file(temp_file)
    temp_file.seek(0)
    #load into joblib
    model=joblib.load(temp_file)
#use the model
model.predict(...)

Option 2 is to use BytesIO like this:

from google.cloud import storage
from sklearn.externals import joblib
from io import BytesIO

storage_client = storage.Client()
bucket_name=<bucket name>
model_bucket='model.joblib'

bucket = storage_client.get_bucket(bucket_name)
#select bucket file
blob = bucket.blob(model_bucket)
#download blob into an in-memory file object
model_file = BytesIO()
blob.download_to_file(model_file)
#load into joblib
model=joblib.load(model_local)

来源：https://stackoverflow.com/questions/51921142/how-to-load-a-model-saved-in-joblib-file-from-google-cloud-storage-bucket

标签

python

scikit-learn

google-cloud-storage

joblib