Keras Text Preprocessing - Saving Tokenizer object to file for scoring

Question

I\'ve trained a sentiment classifier model using Keras library by following the below steps(broadly).

1. Convert Text corpus into sequences using Tokenizer object

Answer 1

I've created the issue https://github.com/keras-team/keras/issues/9289 in the keras Repo. Until the API is changed, the issue has a link to a gist that has code to demonstrate how to save and restore a tokenizer without having the original documents the tokenizer was fit on. I prefer to store all my model information in a JSON file (because reasons, but mainly mixed JS/Python environment), and this will allow for that, even with sort_keys=True