Keras Text Preprocessing - Saving Tokenizer object to file for scoring

Question

I\'ve trained a sentiment classifier model using Keras library by following the below steps(broadly).

1. Convert Text corpus into sequences using Tokenizer object

Answer 1

The most common way is to use either pickle or joblib. Here you have an example on how to use pickle in order to save Tokenizer:

import pickle

# saving
with open('tokenizer.pickle', 'wb') as handle:
    pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)

# loading
with open('tokenizer.pickle', 'rb') as handle:
    tokenizer = pickle.load(handle)