Keras Text Preprocessing - Saving Tokenizer object to file for scoring

Question

I\'ve trained a sentiment classifier model using Keras library by following the below steps(broadly).

1. Convert Text corpus into sequences using Tokenizer object

Answer 1

Tokenizer class has a function to save date into JSON format:

tokenizer_json = tokenizer.to_json()
with io.open('tokenizer.json', 'w', encoding='utf-8') as f:
    f.write(json.dumps(tokenizer_json, ensure_ascii=False))

The data can be loaded using tokenizer_from_json function from keras_preprocessing.text:

with open('tokenizer.json') as f:
    data = json.load(f)
    tokenizer = tokenizer_from_json(data)