What is the Python way to walk a directory tree?

Question

I feel that assigning files, and folders and doing the += [item] part is a bit hackish. Any suggestions? I'm using Python 3.2

from os import *
from os.path import *

def dir_contents(path):
    contents = listdir(path)
    files = []
    folders = []
    for i, item in enumerate(contents):
        if isfile(contents[i]):
            files += [item]
        elif isdir(contents[i]):
            folders += [item]
    return files, folders

Answer 1

Take a look at the os.walk function which returns the path along with the directories and files it contains. That should considerably shorten your solution.

Answer 2

Indeed using

items += [item]

is bad for many reasons...

1. The append method has been made exactly for that (appending one element to the end of a list)

2. You are creating a temporary list of one element just to throw it away. While raw speed should not your first concern when using Python (otherwise you're using the wrong language) still wasting speed for no reason doesn't seem the right thing.

3. You are using a little asymmetry of the Python language... for list objects writing a += b is not the same as writing a = a + b because the former modifies the object in place, while the second instead allocates a new list and this can have a different semantic if the object a is also reachable using other ways. In your specific code this doesn't seem the case but it could become a problem later when someone else (or yourself in a few years, that is the same) will have to modify the code. Python even has a method extend with a less subtle syntax that is specifically made to handle the case in which you want to modify in place a list object by adding at the end the elements of another list.

Also as other have noted seems that your code is trying to do what os.walk already does...

Answer 3

def dir_contents(path):
    files,folders = [],[]
    for p in listdir(path):
        if isfile(p): files.append(p)
        else: folders.append(p)
    return files, folders

Answer 4

Instead of the built-in os.walk and os.path.walk, I use something derived from this piece of code I found suggested elsewhere:

http://code.google.com/p/mylibs/source/browse/lib/Python/MyPyLib/DirectoryStatWalker.py

I won't repaste it here, but it walks the directories recursively and is quite efficient and easy to read.

Answer 5

If you want to recursively iterate through all the files, including all files in the subfolders, I believe this is the best way.

import os

def get_files(input):
    for fd, subfds, fns in os.walk(input):
       for fn in fns:
            yield os.path.join(fd, fn)

## now this will print all full paths

for fn in get_files(fd):
    print(fn)

Answer 6

Since Python 3.4 there is new module pathlib. So to get all dirs and files one can do:

from pathlib import Path

dirs = [str(item) for item in Path(path).iterdir() if item.is_dir()]
files = [str(item) for item in Path(path).iterdir() if item.is_file()]

Answer 7

Try using the append method.

Answer 8

While googling for the same info, I found this question.

I am posting here the smallest, clearest code which I found at http://www.pythoncentral.io/how-to-traverse-a-directory-tree-in-python-guide-to-os-walk/ (rather than just posting the URL, in case of link rot).

The page has some useful info and also points to a few other relevant pages.

# Import the os module, for the os.walk function
import os

# Set the directory you want to start from
rootDir = '.'
for dirName, subdirList, fileList in os.walk(rootDir):
    print('Found directory: %s' % dirName)
    for fname in fileList:
        print('\t%s' % fname)

Answer 9

I've not tested this extensively yet, but I believe this will expand the os.walk generator, join dirnames to all the file paths, and flatten the resulting list; To give a straight up list of concrete files in your search path.

import itertools
import os

def find(input_path):
    return itertools.chain(
        *list(
            list(os.path.join(dirname, fname) for fname in files)
            for dirname, _, files in os.walk(input_path)
        )
    )