Waiting for multiple asynchronous download tasks

Question

I want to download some files, for example 100 files, at the same time. So I decided to add my download threads to a dispatch queue, and GCD will adjust how many threads will run at the same time.

The problem here is: the block in dispatch_async will be completed immediately, because task will run on another thread. So, if urls's length is 100, it will create 100 threads immediately.

var queueDownloadTask = dispatch_queue_create("downloadQueue", nil)

for url in urls {
    dispatch_async(queueDownloadTask) {

        let config = NSURLSessionConfiguration.defaultSessionConfiguration()
        let fileTransferSession = NSURLSession(configuration: config)

        let task = fileTransferSession.downloadTaskWithURL(url, completionHandler: { (responseUrl, response, error) -> Void in
            println("completed")
        })
        task.resume()
    }
}

How can I configure the block in dispatch_async to wait for the download task to complete? I don't want use dispatch_semaphore, because it only allow run one download task at the same time.

Answer 1

To expand on Abhinav's answer, you should:

1. Use dispatch_group_create() to create a group.
2. Call dispatch_group_enter(group) before starting each download task.
3. Call dispatch_group_leave(group) inside the task's completion handler.
4. Then call dispatch_group_notify(group, queue, ^{ ... }) to enqueue a block that will be executed when all the tasks are completed.

You can see an example in this post.

(By the way, it's not true that doing 100 dispatch_asyncs in a row will create 100 threads immediately. The system still retains control over how many threads to use to satisfy the queue. However, your code does not wait for any of the tasks to complete before it returns, nor does it attempt to synchronize between multiple tasks completing.)

Answer 2

In Swift3,

func executeMultiTask() {
     //1. Create group
     let taskGroup = DispatchGroup()

     //2. Enter group
     taskGroup.enter()
     myTask1.execute(completeHandler: {
         // ...
         //3. Leave group
         taskGroup.leave() //< balance with taskGroup.enter()
     })

     /* Add more tasks ...
     //2. Enter group
     taskGroup.enter()
     myTask2.execute(completeHandler: {
         //3. Leave group
         defer {
            // Use `defer` to make sure, `leave()` calls are balanced with `enter()`.
            taskGroup.leave() 
         }
         // ... more
     })
     */

     //4. Notify when all task completed
     taskGroup.notify(queue: DispatchQueue.main, work: DispatchWorkItem(block: { 
         // All tasks are done.
         // ...   
     })

}

Answer 3

A working objective-c example for downloading multi files

- (void) downloadFiles: (NSMutableArray *) fileArray : (NSString *)destParentDir{
    dispatch_group_t serviceGroup = dispatch_group_create();

    for (id fileInfo in fileArray) {
        dispatch_group_enter(serviceGroup);

        NSFileManager *fileManager = [NSFileManager defaultManager];
        NSString *fileName = [fileInfo valueForKey:@"name"];

        //Create SubDirs if needed, note that you need to exclude file name for the dirsString :)
        //[fileManager createDirectoryAtPath:dirsString withIntermediateDirectories:true attributes:nil error:NULL];

        //Download file
        NSURL  *url = [NSURL URLWithString:@"YOUR_FILE_URL"];
        NSData *urlData = [NSData dataWithContentsOfURL:url];
        if(urlData)
        {
            NSString  *localPath = [NSString stringWithFormat:@"%@/%@", destParentDir, fileName];
            [urlData writeToFile:localPath atomically:YES];
        }
        dispatch_group_leave(serviceGroup);
    }

    dispatch_group_notify(serviceGroup, dispatch_get_main_queue(),^{
        NSLog(@"Complete files download");
    });
}

Answer 4

You should use dispatch_group_t. Please refer to Apple documentation to solutionize your case.