Python wget download multiple files at once

问题

Looking for a clean Python Wget solution of downloading multiple files at once.

The url will be always the same :

https://example.com/

So far I can do this :

import wget

print('Beginning file download with wget module')
url = 'https://example.com/new_folder/1.jpg'
wget.download(url)

But i need to download also the -2.jpg, -3.jpg , -4.jpg, -5.jpg and rename the NWZV1WB to something like NEWCODE-1.jpg, NEWCODE-2.jpg...

Also I need to download all content(22).jpg files inside a folder and rename the folder localy to something like NEWCODE, but keep the original name of the files

Here the url also is always the same :

import wget

print('Beginning file download with wget module')
url = 'https://example.com/big/1.jpg' #there's 18 jpg inside
wget.download(url)

What would be best, wget (can't find to many articles about) or requests ? Any help is appreciated.

回答1:

For example:

import wget
import os
import multiprocessing

def run_process(url, output_path):
    wget.download(url, out=output_path)
    # TODO: you can write your rename logic at here using os.rename


if __name__ == '__main__':
    cpus = multiprocessing.cpu_count()
    max_pool_size = 4
    pool = multiprocessing.Pool(cpus if cpus < max_pool_size else max_pool_size)
    base_dir = os.path.dirname(os.path.abspath(__file__))
    target = "NEWCODE"
    prefix_list = ["NWZV1WB", "AWU3JAD", "NW96MRD"]
    download_list = []
    name_list = list(range(1, 23))
    name_list.extend(["zoom_side", "zoom_sole", "zoom_side-thumb"])
    for prefix in prefix_list:
        path = os.path.join(base_dir, prefix)
        if not os.path.exists(path):
            os.mkdir(path)
        if not os.path.isdir(path):
            exit()
        for name in name_list:
            download_list.append(['https://img2.tennis-warehouse.com/360/{p}/{n}.jpg'.format(n=name, p=prefix), path])

    for url, path in download_list: # change here to download other files
        print('Beginning file download with wget module {n}'.format(n=url))
        pool.apply_async(run_process, args=(url, path, ))
    # add your code here to download other files
    pool.close()
    pool.join()
    print("finish")

来源：https://stackoverflow.com/questions/52000950/python-wget-download-multiple-files-at-once

标签

python

download