如何使用 Javascript 下载、压缩和保存多个文件并获得进度？

How to download, zip and save multiple files with Javascript and get progress?

提问by guari

I'm creating a Chrome extension that needs to download multiple files (images and/or videos) from a website. These files may have a huge size, so I want to show the download progress to the user. After some research I found that currently a possible solution might be:

我正在创建一个需要从网站下载多个文件（图像和/或视频）的 Chrome 扩展程序。这些文件可能很大，所以我想向用户显示下载进度。经过一些研究，我发现目前可能的解决方案可能是：

1. Download all the files with XMLHttpRequests.
2. When downloaded, zip all the files into one archive with a JavaScript library (eg. JSZip.js, zip.js).
3. Prompt the user to save the zip with SaveAs dialog.

1. 使用 XMLHttpRequests 下载所有文件。
2. 下载后，使用 JavaScript 库（例如 JSZip.js、zip.js）将所有文件压缩到一个存档中。
3. 使用另存为对话框提示用户保存 zip。

I'm stuck at passage 2), how can I zip the downloaded files?

我被困在第 2) 段，如何压缩下载的文件？

To understand, here is a code sample:

为了理解，这里是一个代码示例：

var fileURLs = ['http://www.test.com/img.jpg',...];
var zip = new JSZip();

var count = 0;
for (var i = 0; i < fileURLs.length; i++){
    var xhr = new XMLHttpRequest();
    xhr.onprogress = calculateAndUpdateProgress;
    xhr.open('GET', fileURLs[i], true);
    xhr.responseType = "blob";
    xhr.onreadystatechange = function () {
        if (xhr.readyState == 4) {
               var blob_url = URL.createObjectURL(response);
            // add downloaded file to zip:
            var fileName = fileURLs[count].substring(fileURLs[count].lastIndexOf('/')+1);
            zip.file(fileName, blob_url); // <- here's one problem

            count++;
            if (count == fileURLs.length){
                // all download are completed, create the zip
                var content = zip.generate();

                // then trigger the download link:
                var zipName = 'download.zip';
                var a = document.createElement('a'); 
                a.href = "data:application/zip;base64," + content;
                a.download = zipName;
                a.click();
            }
        }
    };
    xhr.send();
}

function calculateAndUpdateProgress(evt) {
    if (evt.lengthComputable) {
        // get download progress by performing some average 
        // calculations with evt.loaded, evt.total and the number
        // of file to download / already downloaded
        ...
        // then update the GUI elements (eg. page-action icon and popup if showed)
        ...
    }
}

The upper code generate a downloadable archive containing small corrupted files. There is also an issue with filename sync: blob object do not contains the file name, so If eg. fileURLs[0]takes more time to be downloaded than fileURLs[1]names become wrong (inverted)..

上层代码生成一个包含小损坏文件的可下载存档。文件名同步也有一个问题：blob 对象不包含文件名，所以如果例如。fileURLs[0]下载需要的时间比fileURLs[1]名称错误（倒置）还要多。

NOTE: I know that Chrome has a download API but it's in dev channel so unfortunately it's not a solution now, and I would like to avoid using NPAPI for such a simple task.

注意：我知道 Chrome 有一个下载 API，但它在开发频道中，所以很遗憾它现在不是解决方案，我想避免将 NPAPI 用于这样一个简单的任务。

回答by guari

I was reminded of this question.. since it has no answers yet, I write a possible solution in case it can be useful to someone else:

我想起了这个问题..因为它还没有答案，我写了一个可能的解决方案，以防它对其他人有用：

• as said, the first problem is with passing blob url to jszip (it does not support blobs but it also does not throw any error to notify that and it successfully generates an archive of corrupted files): to correct this, simply pass a base64 string of the data instead of its blob object url;
• the second problem is with file name synchronization: the easiest workaround here is to download one file at a time instead of using parallels xhr requests.

• 如上所述，第一个问题是将 blob url 传递给 jszip（它不支持 blob 但它也不会抛出任何错误来通知它并且它成功地生成了损坏文件的存档）：要纠正这个问题，只需传递一个 base64 字符串数据而不是它的 blob 对象 url；
• 第二个问题是文件名同步：这里最简单的解决方法是一次下载一个文件，而不是使用并行 xhr 请求。

So, the modified upper code can be:

所以，修改后的上层代码可以是：

var fileURLs = ['http://www.test.com/img.jpg',...];
var zip = new JSZip();
var count = 0;

downloadFile(fileURLs[count], onDownloadComplete);


function downloadFile(url, onSuccess) {
    var xhr = new XMLHttpRequest();
    xhr.onprogress = calculateAndUpdateProgress;
    xhr.open('GET', url, true);
    xhr.responseType = "blob";
    xhr.onreadystatechange = function () {
        if (xhr.readyState == 4) {
            if (onSuccess) onSuccess(xhr.response);
}

function onDownloadComplete(blobData){
    if (count < fileURLs.length) {
        blobToBase64(blobData, function(binaryData){
                // add downloaded file to zip:
                var fileName = fileURLs[count].substring(fileURLs[count].lastIndexOf('/')+1);
                zip.file(fileName, binaryData, {base64: true});
                if (count < fileURLs.length -1){
                    count++;
                    downloadFile(fileURLs[count], onDownloadCompleted);
                }
                else {
                    // all files have been downloaded, create the zip
                    var content = zip.generate();

                    // then trigger the download link:        
                    var zipName = 'download.zip';
                    var a = document.createElement('a'); 
                    a.href = "data:application/zip;base64," + content;
                    a.download = zipName;
                    a.click();
                }
            });
    }
}

function blobToBase64(blob, callback) {
    var reader = new FileReader();
    reader.onload = function() {
        var dataUrl = reader.result;
        var base64 = dataUrl.split(',')[1];
        callback(base64);
    };
    reader.readAsDataURL(blob);
}

function calculateAndUpdateProgress(evt) {
    if (evt.lengthComputable) {
        ...
    }
}

Last note, this solution works quite well if you download few and little files (about less than 1MB as whole size for less than 10 files), in other cases JSZip will crash the browser tab when the archive is going to be generated, so it will be a better choice to use a separated thread for compression (a WebWorker, like zip.js does).

最后要注意的是，如果您下载很少的文件（少于 10 个文件的总大小大约小于 1MB），则此解决方案非常有效，在其他情况下，JSZip 将在要生成存档时使浏览器选项卡崩溃，因此它使用单独的线程进行压缩将是更好的选择（一个 WebWorker，就像 zip.js 那样）。

If after that the archive has been generated, the browser still keeps crashing with big files and without reporting any errors, try to trigger the saveAs window without passing binary data, but by passing a blob reference (a.href = URL.createObjectURL(zippedBlobData);where zippedBlobDatais the blob object that refers to the generated archive data);

如果在生成存档之后，浏览器仍然因大文件而崩溃并且没有报告任何错误，请尝试在不传递二进制数据的情况下触发 saveAs 窗口，而是通过传递 blob 引用（a.href = URL.createObjectURL(zippedBlobData);其中zippedBlobData指的是生成的存档数据）；

回答by Lian YuFeng

Based on the @guari code, I tested it locally and applied it to the react application, attaching the code for others' reference.

基于@guari的代码，我在本地进行了测试，应用到react应用中，附上代码供他人参考。

import JSZip from "jszip";
import saveAs from "jszip/vendor/FileSaver.js";

// .......

// download button click event
btnDownloadAudio = record =>{
    let fileURLs = ['https://www.test.com/52f6c50.AMR', 'https://www.test.com/061940.AMR'];
    let count = 0;
    let zip = new JSZip();
    const query = { record, fileURLs, count, zip };
    this.downloadFile(query, this.onDownloadComplete);
}
downloadFile = (query, onSuccess) => {
    const { fileURLs, count, } = query;
    var xhr = new XMLHttpRequest();
    xhr.onprogress = this.calculateAndUpdateProgress;
    xhr.open('GET', fileURLs[count], true);
    xhr.responseType = "blob";
    xhr.onreadystatechange = function (e) {
        if (xhr.readyState == 4) {
            if (onSuccess) onSuccess(query, xhr.response);
        }
    }
    xhr.send();
}
onDownloadComplete = (query, blobData) => {
    let { record, fileURLs, count, zip } = query;
    if (count < fileURLs.length) {
      const _this = this;
      const { audio_list, customer_user_id, } = record;
      this.blobToBase64(blobData, function(binaryData){
        // add downloaded file to zip:
        var sourceFileName = fileURLs[count].substring(fileURLs[count].lastIndexOf('/')+1);
        // convert the source file name to the file name to display
        var displayFileName = audio_list[count].seq + sourceFileName.substring(sourceFileName.lastIndexOf('.'));
        zip.file(displayFileName, binaryData, {base64: true});
        if (count < fileURLs.length -1){
            count++;
            _this.downloadFile({ ...query, count }, _this.onDownloadComplete);
        }
        else {
            // all files have been downloaded, create the zip
            zip.generateAsync({type:"blob"}).then(function(content) {
                // see FileSaver.js
                saveAs(content, `${customer_user_id}.zip`);
            });
        }
      });
    }
}
blobToBase64 = (blob, callback) => {
    var reader = new FileReader();
    reader.onload = function() {
        var dataUrl = reader.result;
        var base64 = dataUrl.split(',')[1];
        callback(base64);
    };
    reader.readAsDataURL(blob);
}
calculateAndUpdateProgress = (evt) => {
    if (evt.lengthComputable) {
        // console.log(evt);
    }
}