ios iOS批量下载多个文件

Downloading multiple files in batches in iOS

提问by ios85

I have an app that right now needs to download hundreds of small PDF's based on the users selection. The problem I am running into is that it is taking a significant amount of time because every time it has to open a new connection. I know that I could use GCD to do an async download, but how would I go about doing this in batches of like 10 files or so. Is there a framework that already does this, or is this something I will have to build my self?

我有一个应用程序，现在需要根据用户的选择下载数百个小 PDF。我遇到的问题是它需要花费大量时间，因为每次它都必须打开一个新连接。我知道我可以使用 GCD 进行异步下载，但是我将如何批量处理 10 个左右的文件。是否有一个框架已经做到了这一点，或者这是我必须建立自己的东西？

回答by Rob

This answer is now obsolete. Now that NSURLConnectionis deprecated and NSURLSessionis now available, that offers better mechanisms for downloading a series of files, avoiding much of the complexity of the solution contemplated here. See my other answerwhich discusses NSURLSession.

这个答案现在已经过时了。现在它NSURLConnection已被弃用并且NSURLSession现在可用，它提供了更好的下载一系列文件的机制，避免了这里考虑的解决方案的大部分复杂性。请参阅我讨论的其他答案NSURLSession。

I'll keep this answer below, for historical purposes.

出于历史目的，我将在下面保留这个答案。

---

---

I'm sure there are lots of wonderful solutions for this, but I wrote a little downloader managerto handle this scenario, where you want to download a bunch of files. Just add the individual downloads to the download manager, and as one finishes, it will kick off the next queued one. You can specify how many you want it to do concurrently (which I default to four), so therefore there's no batching needed. If nothing else, this might provoke some ideas of how you might do this in your own implementation.

我相信有很多很棒的解决方案，但我写了一个小的下载管理器来处理这种情况，你想下载一堆文件。只需将单个下载添加到下载管理器，当一个下载完成后，它将启动下一个排队的下载。您可以指定您希望它同时执行多少（我默认为四个），因此不需要批处理。如果不出意外，这可能会引发一些关于如何在自己的实现中执行此操作的想法。

Note, this offers two advantages:

请注意，这有两个优点：

1. If your files are large, this never holds the entire file in memory, but rather streams it to persistent storage as it's being downloaded. This significantly reduces the memory footprint of the download process.

2. As the files are being downloaded, there are delegate protocols to inform you or the progress of the download.

1. 如果您的文件很大，这永远不会将整个文件保存在内存中，而是在下载时将其流式传输到持久存储中。这显着减少了下载过程的内存占用。

2. 在下载文件时，有委托协议会通知您或下载进度。

I've attempted to describe the classes involved and proper operation on the main page at the Download Manager github page.

我试图在下载管理器 github 页面的主页上描述所涉及的类和正确的操作。

---

---

I should say, though, that this was designed to solve a particular problem, where I wanted to track the progress of downloads of large files as they're being downloaded and where I didn't want to ever hold the entire in memory at one time (e.g., if you're downloading a 100mb file, do you really want to hold that in RAM while downloading?).

不过，我应该说，这是为了解决一个特定问题，我想在下载大文件时跟踪它们的下载进度，并且我不想将整个文件保存在内存中时间（例如，如果您正在下载一个 100mb 的文件，您真的想在下载时将其保存在 RAM 中吗？）。

While my solution solves those problem, if you don't need that, there are far simpler solutions using operation queues. In fact you even hint at this possibility:

虽然我的解决方案解决了这些问题，但如果您不需要它，还有使用操作队列的更简单的解决方案。事实上，你甚至暗示了这种可能性：

I know that I could use GCD to do an async download, but how would I go about doing this in batches of like 10 files or so. ...

我知道我可以使用 GCD 进行异步下载，但是我将如何批量处理 10 个左右的文件。...

I have to say that doing an async download strikes me as the right solution, rather than trying to mitigate the download performance problem by downloading in batches.

我不得不说，异步下载让我觉得是正确的解决方案，而不是试图通过批量下载来缓解下载性能问题。

You talk about using GCD queues. Personally, I'd just create an operation queue so that I could specify how many concurrent operations I wanted, and download the individual files using NSDatamethod dataWithContentsOfURLfollowed by writeToFile:atomically:, making each download it's own operation.

您谈论使用 GCD 队列。就个人而言，我只是创建一个操作队列，以便我可以指定我想要的并发操作数量，并使用NSData方法dataWithContentsOfURL后跟下载各个文件writeToFile:atomically:，使每次下载都是自己的操作。

So, for example, assuming you had an array of URLs of files to download it might be:

因此，例如，假设您有一组要下载的文件 URL，它可能是：

NSOperationQueue *queue = [[NSOperationQueue alloc] init];
queue.maxConcurrentOperationCount = 4;

for (NSURL* url in urlArray)
{
    [queue addOperationWithBlock:^{
        NSData *data = [NSData dataWithContentsOfURL:url];
        NSString *filename = [documentsPath stringByAppendingString:[url lastPathComponent]];
        [data writeToFile:filename atomically:YES];
    }];
}

Nice and simple. And by setting queue.maxConcurrentOperationCountyou enjoy concurrency, while not crushing your app (or the server) with too many concurrent requests.

好看又简单。通过设置queue.maxConcurrentOperationCount您可以享受并发性，同时不会因过多的并发请求而破坏您的应用程序（或服务器）。

And if you need to be notified when the operations are done, you could do something like:

如果您需要在操作完成时收到通知，您可以执行以下操作：

NSOperationQueue *queue = [[NSOperationQueue alloc] init];
queue.maxConcurrentOperationCount = 4;

NSBlockOperation *completionOperation = [NSBlockOperation blockOperationWithBlock:^{
    [[NSOperationQueue mainQueue] addOperationWithBlock:^{
        [self methodToCallOnCompletion];
    }];
}];

for (NSURL* url in urlArray)
{
    NSBlockOperation *operation = [NSBlockOperation blockOperationWithBlock:^{
        NSData *data = [NSData dataWithContentsOfURL:url];
        NSString *filename = [documentsPath stringByAppendingString:[url lastPathComponent]];
        [data writeToFile:filename atomically:YES];
    }];
    [completionOperation addDependency:operation];
}

[queue addOperations:completionOperation.dependencies waitUntilFinished:NO];
[queue addOperation:completionOperation];

This will do the same thing, except it will call methodToCallOnCompletionon the main queue when all the downloads are done.

这将做同样的事情，除了它会methodToCallOnCompletion在所有下载完成后调用主队列。

回答by Rob

By the way, iOS 7 (and Mac OS 10.9) offer URLSessionand URLSessionDownloadTask, which handles this quite gracefully. If you just want to download a bunch of files, you can do something like:

顺便说一下，iOS 7（和 Mac OS 10.9）提供了URLSession和URLSessionDownloadTask，它可以很好地处理这个问题。如果您只想下载一堆文件，您可以执行以下操作：

NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration defaultSessionConfiguration];
NSURLSession              *session       = [NSURLSession sessionWithConfiguration:configuration];

NSString      *documentsPath = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES)[0];
NSFileManager *fileManager   = [NSFileManager defaultManager];

for (NSString *filename in self.filenames) {
    NSURL *url = [baseURL URLByAppendingPathComponent:filename];
    NSURLSessionTask *downloadTask = [session downloadTaskWithURL:url completionHandler:^(NSURL *location, NSURLResponse *response, NSError *error) {
        NSString *finalPath = [documentsPath stringByAppendingPathComponent:filename];

        BOOL success;
        NSError *fileManagerError;
        if ([fileManager fileExistsAtPath:finalPath]) {
            success = [fileManager removeItemAtPath:finalPath error:&fileManagerError];
            NSAssert(success, @"removeItemAtPath error: %@", fileManagerError);
        }

        success = [fileManager moveItemAtURL:location toURL:[NSURL fileURLWithPath:finalPath] error:&fileManagerError];
        NSAssert(success, @"moveItemAtURL error: %@", fileManagerError);

        NSLog(@"finished %@", filename);
    }];
    [downloadTask resume];
}

Perhaps, given that your downloads take a "significant amount of time", you might want them to continue downloading even after the app has gone into the background. If so, you can use backgroundSessionConfigurationrather than defaultSessionConfiguration(though you have to implement the NSURLSessionDownloadDelegatemethods, rather than using the completionHandlerblock). These background sessions are slower, but then again, they happen even if the user has left your app. Thus:

也许，鉴于您的下载需要“大量时间”，您可能希望它们即使在应用程序进入后台后也能继续下载。如果是这样，您可以使用backgroundSessionConfiguration而不是defaultSessionConfiguration（尽管您必须实现NSURLSessionDownloadDelegate方法，而不是使用completionHandler块）。这些后台会话速度较慢，但​​话又说回来，即使用户已离开您的应用程序，它们也会发生。因此：

- (void)startBackgroundDownloadsForBaseURL:(NSURL *)baseURL {
    NSURLSession *session = [self backgroundSession];

    for (NSString *filename in self.filenames) {
        NSURL *url = [baseURL URLByAppendingPathComponent:filename];
        NSURLSessionTask *downloadTask = [session downloadTaskWithURL:url];
        [downloadTask resume];
    }
}

- (NSURLSession *)backgroundSession {
    static NSURLSession *session = nil;

    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration backgroundSessionConfiguration:kBackgroundId];
        session = [NSURLSession sessionWithConfiguration:configuration delegate:self delegateQueue:[NSOperationQueue mainQueue]];
    });

    return session;
}

#pragma mark - NSURLSessionDownloadDelegate

- (void)URLSession:(NSURLSession *)session downloadTask:(NSURLSessionDownloadTask *)downloadTask didFinishDownloadingToURL:(NSURL *)location {
    NSString *documentsPath    = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES)[0];
    NSString *finalPath        = [documentsPath stringByAppendingPathComponent:[[[downloadTask originalRequest] URL] lastPathComponent]];
    NSFileManager *fileManager = [NSFileManager defaultManager];

    BOOL success;
    NSError *error;
    if ([fileManager fileExistsAtPath:finalPath]) {
        success = [fileManager removeItemAtPath:finalPath error:&error];
        NSAssert(success, @"removeItemAtPath error: %@", error);
    }

    success = [fileManager moveItemAtURL:location toURL:[NSURL fileURLWithPath:finalPath] error:&error];
    NSAssert(success, @"moveItemAtURL error: %@", error);
}

- (void)URLSession:(NSURLSession *)session downloadTask:(NSURLSessionDownloadTask *)downloadTask didResumeAtOffset:(int64_t)fileOffset expectedTotalBytes:(int64_t)expectedTotalBytes {
    // Update your UI if you want to
}

- (void)URLSession:(NSURLSession *)session downloadTask:(NSURLSessionDownloadTask *)downloadTask didWriteData:(int64_t)bytesWritten totalBytesWritten:(int64_t)totalBytesWritten totalBytesExpectedToWrite:(int64_t)totalBytesExpectedToWrite {
    // Update your UI if you want to
}

- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task didCompleteWithError:(NSError *)error {
    if (error)
        NSLog(@"%s: %@", __FUNCTION__, error);
}

#pragma mark - NSURLSessionDelegate

- (void)URLSession:(NSURLSession *)session didBecomeInvalidWithError:(NSError *)error {
    NSLog(@"%s: %@", __FUNCTION__, error);
}

- (void)URLSessionDidFinishEventsForBackgroundURLSession:(NSURLSession *)session {
    AppDelegate *appDelegate = (id)[[UIApplication sharedApplication] delegate];
    if (appDelegate.backgroundSessionCompletionHandler) {
        dispatch_async(dispatch_get_main_queue(), ^{
            appDelegate.backgroundSessionCompletionHandler();
            appDelegate.backgroundSessionCompletionHandler = nil;
        });
    }
}

By the way, this assumes your app delegate has a backgroundSessionCompletionHandlerproperty:

顺便说一下，这假设您的应用程序委托有一个backgroundSessionCompletionHandler属性：

@property (copy) void (^backgroundSessionCompletionHandler)();

And that the app delegate will set that property if the app was awaken to handle URLSessionevents:

如果应用程序被唤醒以处理URLSession事件，应用程序委托将设置该属性：

- (void)application:(UIApplication *)application handleEventsForBackgroundURLSession:(NSString *)identifier completionHandler:(void (^)())completionHandler {
    self.backgroundSessionCompletionHandler = completionHandler;
}

For an Apple demonstration of the background NSURLSessionsee the Simple Background Transfersample.

有关背景的 Apple 演示，NSURLSession请参阅简单背景传输示例。

回答by rmaddy

If all of the PDFs are coming from a server you control then one option would be to have a single request pass a list of files you want (as query parameters on the URL). Then your server could zip up the requested files into a single file.

如果所有 PDF 都来自您控制的服务器，那么一种选择是让单个请求传递您想要的文件列表（作为 URL 上的查询参数）。然后您的服务器可以将请求的文件压缩到一个文件中。

This would cut down on the number of individual network requests you need to make. Of course you need to update your server to handle such a request and your app needs to unzip the returned file. But this is much more efficient than making lots of individual network requests.

这将减少您需要发出的单个网络请求的数量。当然，您需要更新服务器以处理此类请求，并且您的应用程序需要解压缩返回的文件。但这比发出大量单独的网络请求要有效得多。

回答by jsd

Use an NSOperationQueue and make each download a separate NSOperation. Set the maximum concurrent operations property on your queue to however many downloads you want to be able to run simultaneously. I'd keep it in the 4-6 range personally.

使用 NSOperationQueue 并使每次下载成为一个单独的 NSOperation。将队列上的最大并发操作数属性设置为您希望能够同时运行的下载次数。我个人会将其保持在 4-6 范围内。

Here's a good blog post that explains how to make concurrent operations. http://www.dribin.org/dave/blog/archives/2009/05/05/concurrent_operations/

这是一篇很好的博客文章，解释了如何进行并发操作。 http://www.dribin.org/dave/blog/archives/2009/05/05/concurrent_operations/

回答by Kamen Dobrev

What came as a big surprise is how slow dataWithContentsOfURL is when downloading multiple files!

令人惊讶的是 dataWithContentsOfURL 在下载多个文件时有多慢！

To see it by yourself run the following example: (you don't need the downloadQueue for downloadTaskWithURL, its there just for easier comparison)

要自己查看它，请运行以下示例：（您不需要 downloadTaskWithURL 的 downloadQueue，它只是为了便于比较）

- (IBAction)downloadUrls:(id)sender {
    [[NSOperationQueue new] addOperationWithBlock:^{
        [self download:true];
        [self download:false];
    }];
}

-(void) download:(BOOL) slow
{
       double startTime = CACurrentMediaTime();
        NSURLSessionConfiguration* config = [NSURLSessionConfiguration defaultSessionConfiguration];
        static NSURLSession* urlSession;

        if(urlSession == nil)
            urlSession = [NSURLSession sessionWithConfiguration:config delegate:nil delegateQueue:nil];

       dispatch_group_t syncGroup = dispatch_group_create();
       NSOperationQueue* downloadQueue = [NSOperationQueue new];
       downloadQueue.maxConcurrentOperationCount = 10;

       NSString* baseUrl = @"https://via.placeholder.com/468x60?text="; 
       for(int i = 0;i < 100;i++) {
           NSString* urlString = [baseUrl stringByAppendingFormat:@"image%d", i];
           dispatch_group_enter(syncGroup);
           NSURL  *url = [NSURL URLWithString:urlString];
           [downloadQueue addOperationWithBlock:^{
               if(slow) {
                   NSData *urlData = [NSData dataWithContentsOfURL:url];
                   dispatch_group_leave(syncGroup);
                   //NSLog(@"downloaded: %@", urlString);
               }
               else {
                   NSURLSessionDownloadTask* task = [urlSession downloadTaskWithURL:url completionHandler:^(NSURL * _Nullable location, NSURLResponse * _Nullable response, NSError * _Nullable error) {
                   //NSLog(@"downloaded: %@", urlString);
                   dispatch_group_leave(syncGroup);
                   }];[task resume];
               }
           }];
       }

       dispatch_group_wait(syncGroup, DISPATCH_TIME_FOREVER);

       double endTime = CACurrentMediaTime();
       NSLog(@"Download time:%.2f", (endTime - startTime));
}

回答by Mundi

There is nothing to "build". Just loop through the next 10 files each time in 10 threads and get the next file when a thread finishes.

没有什么可以“建造”的。只需在 10 个线程中每次循环遍历接下来的 10 个文件，并在线程完成时获取下一个文件。