Searching for a string in a large text file - profiling various methods in python

Question

This question has been asked many times. After spending some time reading the answers, I did some quick profiling to try out the various methods mentioned previously...

Answer 1

I would guess many of the paths start out the same on DMOZ. You should use a trie data structure and store the individual characters on nodes.

Tries have O(m) lookup time (where m is the key length) also save a lot of space, when saving large dictionaries or tree like data.

You could also store path parts on nodes to reduce node count — this is called Patricia Trie. But that makes the lookup slower by the average string length comparison time. See SO question Trie (Prefix Tree) in Python for more info about implementations.

There are a couple of trie implementations on Python Package Index, but they are not very good. I have written one in Ruby and in Common Lisp, which is especially well suited for this task – if you ask nicely, I could maybe publish it as open source... :-)