如何使用 Java 获取上传到 Amazon S3 的文件的进度状态
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/11908385/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How to get the progress status of the file uploaded to Amazon S3 using Java
提问by Krushna
I'm uploading multiple files to Amazon S3 using Java.
我正在使用 Java 将多个文件上传到 Amazon S3。
The code I'm using is as follows:
我使用的代码如下：
MultipartHttpServletRequest multipartRequest = (MultipartHttpServletRequest) request;
MultiValueMap < String,
MultipartFile > map = multipartRequest.getMultiFileMap();
try {
if (map != null) {
for (String filename: map.keySet()) {
List < MultipartFile > fileList = map.get(filename);
incrPercentge = 100 / fileList.size();
request.getSession().setAttribute("incrPercentge", incrPercentge);
for (MultipartFile mpf: fileList) {
/*
* custom input stream wrap to original input stream to get
* the progress
*/
ProgressInputStream inputStream = new ProgressInputStream("test", mpf.getInputStream(), mpf.getBytes().length);
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType(mpf.getContentType());
String key = Util.getLoginUserName() + "/" + mpf.getOriginalFilename();
PutObjectRequest putObjectRequest = new PutObjectRequest(
Constants.S3_BUCKET_NAME, key, inputStream, metadata).withStorageClass(StorageClass.ReducedRedundancy);
PutObjectResult response = s3Client.putObject(putObjectRequest);
}
}
}
} catch(Exception e) {
e.printStackTrace();
}
I have to create the custom input stream to get the number byte consumed by Amazon S3. I got that idea from the question here: Upload file or InputStream to S3 with a progress callback
我必须创建自定义输入流以获取 Amazon S3 使用的字节数。我从这里的问题中得到了这个想法：Upload file or InputStream to S3 with a progress callback
My ProgressInputStreamclass code is as follows:
我的ProgressInputStream类代码如下：
package com.spectralnetworks.net.util;
import java.io.IOException;
import java.io.InputStream;
import org.apache.commons.vfs.FileContent;
import org.apache.commons.vfs.FileSystemException;
public class ProgressInputStream extends InputStream {
private final long size;
private long progress,
lastUpdate = 0;
private final InputStream inputStream;
private final String name;
private boolean closed = false;
public ProgressInputStream(String name, InputStream inputStream, long size) {
this.size = size;
this.inputStream = inputStream;
this.name = name;
}
public ProgressInputStream(String name, FileContent content)
throws FileSystemException {
this.size = content.getSize();
this.name = name;
this.inputStream = content.getInputStream();
}
@Override
public void close() throws IOException {
super.close();
if (closed) throw new IOException("already closed");
closed = true;
}
@Override
public int read() throws IOException {
int count = inputStream.read();
if (count > 0) progress += count;
lastUpdate = maybeUpdateDisplay(name, progress, lastUpdate, size);
return count;
}@Override
public int read(byte[] b, int off, int len) throws IOException {
int count = inputStream.read(b, off, len);
if (count > 0) progress += count;
lastUpdate = maybeUpdateDisplay(name, progress, lastUpdate, size);
return count;
}
/**
* This is on reserach to show a progress bar
* @param name
* @param progress
* @param lastUpdate
* @param size
* @return
*/
static long maybeUpdateDisplay(String name, long progress, long lastUpdate, long size) {
/* if (Config.isInUnitTests()) return lastUpdate;
if (size < B_IN_MB/10) return lastUpdate;
if (progress - lastUpdate > 1024 * 10) {
lastUpdate = progress;
int hashes = (int) (((double)progress / (double)size) * 40);
if (hashes > 40) hashes = 40;
String bar = StringUtils.repeat("#",
hashes);
bar = StringUtils.rightPad(bar, 40);
System.out.format("%s [%s] %.2fMB/%.2fMB\r",
name, bar, progress / B_IN_MB, size / B_IN_MB);
System.out.flush();
}*/
System.out.println("name " + name + " progress " + progress + " lastUpdate " + lastUpdate + " " + "sie " + size);
return lastUpdate;
}
}
But this is not working properly. It is printing immediately up to the file size as follows:
但这不能正常工作。它立即打印到文件大小，如下所示：
name test progress 4096 lastUpdate 0 sie 30489
name test progress 8192 lastUpdate 0 sie 30489
name test progress 12288 lastUpdate 0 sie 30489
name test progress 16384 lastUpdate 0 sie 30489
name test progress 20480 lastUpdate 0 sie 30489
name test progress 24576 lastUpdate 0 sie 30489
name test progress 28672 lastUpdate 0 sie 30489
name test progress 30489 lastUpdate 0 sie 30489
name test progress 30489 lastUpdate 0 sie 30489
And the actual uploading is taking more time (more than 10 times after printing the lines).
实际上传需要更多时间（打印行后超过 10 次）。
What I should do so that I can get a true upload status?
我应该怎么做才能获得真正的上传状态？
回答by Krushna
I got the answer of my questions the best way get the true progress status by using below code
我得到了我的问题的答案，这是使用以下代码获得真实进度状态的最佳方式
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType(mpf.getContentType());
String key = Util.getLoginUserName() + "/"
+ mpf.getOriginalFilename();
metadata.setContentLength(mpf.getSize());
PutObjectRequest putObjectRequest = new PutObjectRequest(
Constants.S3_BUCKET_NAME, key, mpf.getInputStream(),
metadata)
.withStorageClass(StorageClass.ReducedRedundancy);
putObjectRequest.setProgressListener(new ProgressListener() {
@Override
public void progressChanged(ProgressEvent progressEvent) {
System.out.println(progressEvent
.getBytesTransfered()
+ ">> Number of byte transfered "
+ new Date());
progressEvent.getBytesTransfered();
double totalByteRead = request
.getSession().getAttribute(
Constants.TOTAL_BYTE_READ) != null ? (Double) request
.getSession().getAttribute(Constants.TOTAL_BYTE_READ) : 0;
totalByteRead += progressEvent.getBytesTransfered();
request.getSession().setAttribute(Constants.TOTAL_BYTE_READ, totalByteRead);
System.out.println("total Byte read "+ totalByteRead);
request.getSession().setAttribute(Constants.TOTAL_PROGRESS, (totalByteRead/size)*100);
System.out.println("percentage completed >>>"+ (totalByteRead/size)*100);
if (progressEvent.getEventCode() == ProgressEvent.COMPLETED_EVENT_CODE) {
System.out.println("completed ******");
}
}
});
s3Client.putObject(putObjectRequest);
The problem with my previous code was , I was not setting the content length in meta data so i was not getting the true progress status. The below line is copy from PutObjectRequest class API
我之前代码的问题是，我没有在元数据中设置内容长度，所以我没有得到真正的进度状态。下面的行是从 PutObjectRequest 类 API 复制的
Constructs a new PutObjectRequest object to upload a stream of data to the specified bucket and key. After constructing the request, users may optionally specify object metadata or a canned ACL as well.
构造一个新的 PutObjectRequest 对象以将数据流上传到指定的存储桶和键。构建请求后，用户还可以选择指定对象元数据或预制 ACL。
Content length for the data stream must be specified in the object metadata parameter; Amazon S3 requires it be passed in before the data is uploaded. Failure to specify a content length will cause the entire contents of the input stream to be buffered locally in memory so that the content length can be calculated, which can result in negative performance problems.
数据流的内容长度必须在对象元数据参数中指定；Amazon S3 要求在上传数据之前传入它。未指定内容长度将导致输入流的全部内容在本地缓存在内存中，以便可以计算内容长度，这可能会导致负面的性能问题。
回答by Eli Algranti
I going to assume you are using the AWS SDK for Java.
我假设您使用的是适用于 Java 的 AWS 开发工具包。
Your code is working as it should: It shows read is being called with 4K being read each time. Your idea (updated in the message) is also correct: The AWS SDK provides ProgressListener as a way to inform the application of progress in the upload.
您的代码正在正常工作：它显示正在调用 read，每次读取 4K。您的想法（在消息中更新）也是正确的：AWS 开发工具包提供了 ProgressListener 作为通知应用程序上传进度的一种方式。
The "problem" is in the implementation of the AWS SDK it is buffering more than the ~30K size of your file (I'm going to assume it's 64K) so you're not getting any progress reports.
“问题”出在 AWS 开发工具包的实现中，它缓冲的文件大小超过了大约 30K（我假设它是 64K），因此您没有收到任何进度报告。
Try to upload a bigger file (say 1M) and you'll see both methods give you better results, after all with today's network speeds reporting the progress on a 30K file is not even worth it.
尝试上传一个更大的文件（比如 1M），你会发现这两种方法都能给你更好的结果，毕竟以今天的网络速度报告 30K 文件的进度甚至不值得。
If you want better control you could implement the upload yourself using the S3 REST interface(which is what the AWS Java SDK ultimately uses) it is not very difficult, but it is a bit of work. If you want to go this route I recommend finding an example for computing the session authorization token instead of doing it yourself(sorry my search foo is not strong enough for a link to actual sample code right now.) However once you go to all that trouble you'll find that you actually want to have a 64K buffer on the socket stream to ensure maximum throughput in a fast network (which is probably why the AWS Java SDK behaves as it does.)
如果您想要更好的控制，您可以使用S3 REST 接口（这是 AWS Java SDK 最终使用的接口）自己实现上传，这不是很困难，但需要一些工作。如果你想走这条路，我建议你找一个例子来计算会话授权令牌，而不是自己做（对不起，我的搜索 foo 不够强大，现在无法链接到实际示例代码。）但是，一旦你完成了所有这些麻烦您会发现您实际上希望在套接字流上有一个 64K 缓冲区，以确保快速网络中的最大吞吐量（这可能是 AWS Java SDK 表现如此的原因。）
