问题
I am trying to run a command to download 3000 files in parallel. I am using Cygwin + Windows.
Downloading a single file via WGET in terminal :
wget --no-check-certificate --content-disposition --load-cookies cookies.txt \ -p https://username:password@website.webpage.com/folder/document/download/1?type=file
allows me to download the file with ID 1 singularly, in the correct format (as long as --content-disposition is in the command).
I iterate over this REST API call to download the entire folder (3000 files). This works OK, but is quite slow.
FOR /L %i in (0,1,3000) do wget --no-check-certificate --content-disposition --load-cookies cookies.txt \ -p https://username:password@website.webpage.com/folder/document/download/%i?type=file
Now I am trying to run the program in Cygwin, in parallel.
seq 3000 | parallel -j 200 wget --no-check-certificate --content-disposition --load-cookies cookies.txt -p https://username:password@website.webpage.com/folder/document/download/{}?type=file
It runs, but the file-name and format is lost (instead of "index.html", for example, we may get "4@type=file" as the file-name).
Is there a way for me to fix this?
回答1:
It is unclear what you would like them named. Let us assume you want them named: index.[1-3000].html
seq 3000 | parallel -j 200 wget -O index.{}.html --no-check-certificate --content-disposition --load-cookies cookies.txt -p https://username:password@website.webpage.com/folder/document/download/{}?type=file
My guess is that it is caused by --content-disposition being experimental, and the wget used by CygWin may be older than the wget used by the FOR loop. To check that run:
wget --version
in CygWin and outside CygWin (ie. where you would run the FOR loop).
来源：https://stackoverflow.com/questions/29942855/wget-ignoring-content-disposition