How to quickly get the collection of words in a corpus (with nltk)?

Question

I would like to quickly build a word look-up table for a corpus with nltk. Below is what I am doing:

1. Read raw text: file=open("corpus","r").read().decode('utf-8')
2. Use a=nltk.word_tokenize(file) to get all tokens;
3. Use set(a) to get unique tokens, and covert it back to a list.

Is this the right way of doing this task?

Answer 1

Try:

import time
from collections import Counter

from nltk import FreqDist
from nltk.corpus import brown
from nltk import word_tokenize

def time_uniq(maxchar):
    # Let's just take the first 10000 characters.
    words = brown.raw()[:maxchar] 

    # Time to tokenize
    start = time.time()
    words = word_tokenize(words)
    print time.time() - start

    # Using collections.Counter
    start = time.time()
    x = Counter(words)
    uniq_words = x.keys()
    print time.time() - start

    # Using nltk.FreqDist
    start = time.time()
    FreqDist(words)
    uniq_words = x.keys()
    print time.time() - start

    # If you don't need frequency info, use set()
    start = time.time()
    uniq_words = set(words)
    print time.time() - start

[out]:

~$ python test.py 
0.0413908958435
0.000495910644531
0.000432968139648
9.3936920166e-05

0.10734796524
0.00458407402039
0.00439405441284
0.00084400177002

1.12890005112
0.0492491722107
0.0490930080414
0.0100378990173

To load your own corpus file (assuming that your file is small enough to fit into the RAM):

from collections import Counter
from nltk import FreqDist, word_tokenize

with open('myfile.txt', 'r') as fin:
    # Using Counter.
    x = Counter(word_tokenize(fin.read()))
    uniq = x.keys()
    # Using FreqDist
    x = Counter(word_tokenize(fin.read()))
    uniq = x.keys()
    # Using Set
    uniq = set(word_tokenize(fin.read()))

If file is too big, possibly you want to process the file one line at a time:

from collections import Counter
from nltk import FreqDist, word_tokenize

from nltk.corpus import brown

# Using Counter.
x = Counter()
with open('myfile.txt', 'r') as fin:
    for line in fin.split('\n'):
        x.update(word_tokenize(line))
uniq = x.keys()

# Using Set.
x = set()
with open('myfile.txt', 'r') as fin:
    for line in fin.split('\n'):
        x.update(word_tokenize(line))
uniq = x.keys()