aria2c - Any way to keep only list of failed downloads?

ε祈祈猫儿з 提交于 2021-02-10 17:41:03

问题


I am using aria2c to download a quite large list of urls (~6000) organized in a text file.

Based on this gist, I am using the following script to download all the files:

#!/bin/bash
aria2c -j5 -i list.txt -c --save-session out.txt
has_error=`wc -l < out.txt`

while [ $has_error -gt 0 ]
do
  echo "still has $has_error errors, rerun aria2 to download ..."
  aria2c -j5 -i list.txt -c --save-session out.txt
  has_error=`wc -l < out.txt`
  sleep 10
done

### PS: one line solution, just loop 1000 times
###         seq 1000 | parallel -j1 aria2c -i list.txt -c

which saves all the aria2c output in a text file, and in case of at least one download error, tries to download all urls again.

The problem is: since my url list is so large, this becomes pretty inefficient. If 30 files result in download error (say, because of a server timeout), then the whole list will be looped over again 30 times.

So, the question is: is there any way to tell aria2c to save only the failed downloads, and then try to re-download only those files?

来源:https://stackoverflow.com/questions/63821612/aria2c-any-way-to-keep-only-list-of-failed-downloads

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!