node.js 使用节点js的S3文件上传流
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/21657700/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
S3 file upload stream using node js
提问by Janak Kansal
I am trying to find some solution to stream file on amazon S3 using node js server with requirements:
我正在尝试使用具有要求的节点 js 服务器找到一些解决方案来在 amazon S3 上流式传输文件：
- Don't store temp file on server or in memory. But up-to some limit not complete file, buffering can be used for uploading.
- No restriction on uploaded file size.
- Don't freeze server till complete file upload because in case of heavy file upload other request's waiting time will unexpectedly increase.
- 不要将临时文件存储在服务器或内存中。但在一定限度内不完整的文件，可以使用缓冲上传。
- 上传的文件大小没有限制。
- 在文件上传完成之前不要冻结服务器，因为在大量文件上传的情况下，其他请求的等待时间会意外增加。
I don't want to use direct file upload from browser because S3 credentials needs to share in that case. One more reason to upload file from node js server is that some authentication may also needs to apply before uploading file.
我不想使用从浏览器直接上传文件，因为在这种情况下需要共享 S3 凭据。从 node js 服务器上传文件的另一个原因是在上传文件之前可能还需要应用一些身份验证。
I tried to achieve this using node-multiparty. But it was not working as expecting. You can see my solution and issue at https://github.com/andrewrk/node-multiparty/issues/49. It works fine for small files but fails for file of size 15MB.
我尝试使用 node-multiparty 来实现这一点。但它并没有像预期的那样工作。您可以在https://github.com/andrewrk/node-multiparty/issues/49查看我的解决方案和问题。它适用于小文件，但不适用于 15MB 的文件。
Any solution or alternative ?
任何解决方案或替代方案？
回答by Johann Philipp Strathausen
You can now use streaming with the official Amazon SDK for nodejsand what's even more awesome, you finally can do so without knowing the file size in advance. Simply pass the stream as the Body:
您现在可以使用用于 nodejs的官方 Amazon SDK 进行流式处理，更棒的是，您终于可以在不提前知道文件大小的情况下这样做。只需将流传递为Body：
var fs = require('fs');
var zlib = require('zlib');
var body = fs.createReadStream('bigfile').pipe(zlib.createGzip());
var s3obj = new AWS.S3({params: {Bucket: 'myBucket', Key: 'myKey'}});
s3obj.upload({Body: body})
.on('httpUploadProgress', function(evt) { console.log(evt); })
.send(function(err, data) { console.log(err, data) });
回答by Yaroslav Pogrebnyak
Give https://www.npmjs.org/package/streaming-s3a try.
给https://www.npmjs.org/package/streaming-s3一试。
I used it for uploading several big files in parallel (>500Mb), and it worked very well. It very configurable and also allows you to track uploading statistics. You not need to know total size of the object, and nothing is written on disk.
我用它来并行上传几个大文件（>500Mb），效果很好。它非常可配置，还允许您跟踪上传统计信息。您不需要知道对象的总大小，也不会在磁盘上写入任何内容。
回答by Daveee
I'm using the s3-upload-streammodule in a working project here.
我在这里的一个工作项目中使用s3-upload-stream模块。
There is also some good examples from @raynos in his http-frameworkrepository.
@raynos 在他的http-framework存储库中也有一些很好的例子。
回答by Harshavardhana
Alternatively you can look at - https://github.com/minio/minio-js. It has minimal set of abstracted API's implementing most commonly used S3 calls.
或者，您可以查看 - https://github.com/minio/minio-js。它具有最少的抽象 API 集，用于实现最常用的 S3 调用。
Here is an example of streaming upload.
这是流式上传的示例。
$ npm install minio
$ cat >> put-object.js << EOF
var Minio = require('minio')
var fs = require('fs')
// find out your s3 end point here:
// http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
var s3Client = new Minio({
url: 'https://<your-s3-endpoint>',
accessKey: 'YOUR-ACCESSKEYID',
secretKey: 'YOUR-SECRETACCESSKEY'
})
var outFile = fs.createWriteStream('your_localfile.zip');
var fileStat = Fs.stat(file, function(e, stat) {
if (e) {
return console.log(e)
}
s3Client.putObject('mybucket', 'hello/remote_file.zip', 'application/octet-stream', stat.size, fileStream, function(e) {
return console.log(e) // should be null
})
})
EOF
putObject() here is a fully managed single function call for file sizes over 5MB it automatically does multipart internally. You can resume a failed upload as well and it will start from where its left off by verifying previously upload parts.
putObject() 这里是一个完全托管的单一函数调用，文件大小超过 5MB，它会在内部自动执行多部分。您也可以恢复失败的上传，它会通过验证以前上传的部分从停止的地方开始。
Additionally this library is also isomorphic, can be used in browsers as well.
此外，这个库也是同构的，也可以在浏览器中使用。
回答by mattdlockyer
If it helps anyone I was able to stream from the client to s3 successfully (without memory or disk storage):
如果它帮助我能够成功地从客户端流式传输到 s3 的任何人（没有内存或磁盘存储）：
https://gist.github.com/mattlockyer/532291b6194f6d9ca40cb82564db9d2a
https://gist.github.com/mattlockyer/532291b6194f6d9ca40cb82564db9d2a
The server endpoint assumes reqis a stream object, I sent a File object from the client which modern browsers can send as binary data and added file info set in the headers.
服务器端点假定req是一个流对象，我从客户端发送了一个 File 对象，现代浏览器可以将其作为二进制数据发送，并在标头中添加了文件信息集。
const fileUploadStream = (req, res) => {
//get "body" args from header
const { id, fn } = JSON.parse(req.get('body'));
const Key = id + '/' + fn; //upload to s3 folder "id" with filename === fn
const params = {
Key,
Bucket: bucketName, //set somewhere
Body: req, //req is a stream
};
s3.upload(params, (err, data) => {
if (err) {
res.send('Error Uploading Data: ' + JSON.stringify(err) + '\n' + JSON.stringify(err.stack));
} else {
res.send(Key);
}
});
};
Yes putting the file info in the headers breaks convention but if you look at the gist it's much cleaner than anything else I found using streaming libraries or multer, busboy etc...
是的，将文件信息放在标题中打破了惯例，但如果你看一下要点，它比我使用流媒体库或multer、busboy 等发现的任何其他东西都要干净得多......
+1 for pragmatism and thanks to @SalehenRahman for his help.
+1 务实，感谢@SalehenRahman 的帮助。
