Keras Tokenizer num_words doesn't seem to work

Question

>>> t = Tokenizer(num_words=3)
>>> l = [\"Hello, World! This is so&#$ fantastic!\", \"There is no other world like this one\"]
>>> t.f         


        

Answer 1

Limiting num_words to a small number (eg, 3) has no effect on fit_on_texts outputs such as word_index, word_counts, word_docs. It does have effect on texts_to_matrix. The resulting matrix will have num_words (3) columns.

>>> t = Tokenizer(num_words=3)
>>> l = ["Hello, World! This is so&#$ fantastic!", "There is no other world like this one"]
>>> t.fit_on_texts(l)
>>> print(t.word_index)
{'world': 1, 'this': 2, 'is': 3, 'hello': 4, 'so': 5, 'fantastic': 6, 'there': 7, 'no': 8, 'other': 9, 'like': 10, 'one': 11}

>>> t.texts_to_matrix(l, mode='count')
array([[0., 1., 1.],       
       [0., 1., 1.]])