Recursively copying files with progress

Question

I've seen questions asked here before about Python and copying files, but I have a different scenario to deal with.

I'm almost done with a Linux distro installer I've been working on, and now all it needs to do is copy the files over to the destination partition. As most distro installers have a progress bar, I was hoping to add one too.

Right now, I'm using PyQt4 and my code looks like this:

self.status('Counting files...')
self.count = int(check_output(['-c', 'find /opt/linux/work/root-image/ -type f | wc -l'], stderr = PIPE, shell = True))

self.status('Copying files...')

i = 0

for root, dirs, files in os.walk('/opt/linux/work/root-image/'):
  for file in files:
    i += 1
    f = os.path.join(root, file)

    try:
      os.system('mkdir -p /tmp/foo' + os.path.split(f)[0])
    except:
      pass

    os.system('cp ' + f + ' /tmp/foo' + f)

    if i % 100 == 0:
      self.emit(SIGNAL('progress(int)'), int(100.0 * float(i) / float(self.count)))

self.status('Done...')

It's quite inefficient because of the progress bar. The whole image is 2.1GB, and it takes the script a really long time to copy the files over. Much longer than a simple cp -r.

Is there any efficient way to do this? For single-file copy progressbars, all you do is read little chunks at a time, but I have no idea how to do that for a directory with 91,489 files.

Any help would be helpful. Thanks!

Answer 1

You could try using shutil.copy to copy files instead of calling out to the OS using os.system (which creates a separate process). You can also use os.mkdir to create new directories. However, are you sure that it is slow because of the progress bar and not something else?