Is there a way to efficiently yield every file in a directory containing millions of files?

Question

I\'m aware of os.listdir, but as far as I can gather, that gets all the filenames in a directory into memory, and then returns the list. What I want, is a way t

Answer 1

The glob module Python from 2.5 onwards has an iglob method which returns an iterator. An iterator is exactly for the purposes of not storing huge values in memory.

glob.iglob(pathname)
Return an iterator which yields the same values as glob() without
actually storing them all simultaneously.

For example:

import glob
for eachfile in glob.iglob('*'):
    # act upon eachfile