java 在 S3 中迭代对象时出现“ConnectionPoolTimeoutException”
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/17782937/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
"ConnectionPoolTimeoutException" when iterating objects in S3
提问by Fgblanch
I've been working for some time with aws java API with not so many problems. Currently I'm using the library 1.5.2 version.
我已经使用 aws java API 工作了一段时间，没有那么多问题。目前我正在使用库 1.5.2 版本。
When I'm iterating the objects inside a folder with the following code:
当我使用以下代码迭代文件夹内的对象时：
AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(MyClass.class.getResourceAsStream("AwsCredentials.properties")));
String s3Key = "folder1/folder2";
String bucketName = Constantes.S3_BUCKET;
String key = s3Key +"/input_chopped/";
ObjectListing current = s3.listObjects(new ListObjectsRequest()
.withBucketName(bucketName)
.withPrefix(key));
boolean siguiente = true;
while (siguiente) {
siguiente &= current.isTruncated();
contador += current.getObjectSummaries().size();
for (S3ObjectSummary objectSummary : current.getObjectSummaries()) {
S3Object object = s3.getObject(new GetObjectRequest(bucketName, objectSummary.getKey()));
System.out.println(object.getKey());
}
current=s3.listNextBatchOfObjects(current);
}
Gist: Link: https://gist.github.com/fgblanch/6038699I'm getting the following exception:
要点：链接：https: //gist.github.com/fgblanch/6038699我收到以下异常：
INFO (AmazonHttpClient.java:358) - Unable to execute HTTP request: Timeout waiting for connection from pool
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:232)
at org.apache.http.impl.conn.PoolingClientConnectionManager.getConnection(PoolingClientConnectionManager.java:199)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:315)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:199)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2994)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:918)
at com.madiva.segmentacion.tests.ListaS3.main(ListaS3.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caught an AmazonClientException, which means the client encountered a serious internal problem while trying to communicate with S3, such as not being able to access the network.
Error Message: Unable to execute HTTP request: Timeout waiting for connection from pool
Any idea how to avoid this error. It only happens in folders with a number of object , in this case there were 463 files inside. Thanks
任何想法如何避免这个错误。它只发生在具有多个 object 的文件夹中，在这种情况下，里面有 463 个文件。谢谢
回答by Fgblanch
I've found that S3Object opens a connection for each object. That are not liberated even if the object is garbage collected so it is needed to execute object.close(), in order to liberate the connection to the pool.
我发现 S3Object 为每个对象打开一个连接。即使对象被垃圾收集也不会被释放，因此需要执行 object.close() 以释放与池的连接。
So the corrected code would be:
所以更正后的代码是：
for (S3ObjectSummary objectSummary : current.getObjectSummaries()) {
S3Object object = s3.getObject(new GetObjectRequest(bucketName, objectSummary.getKey()));
System.out.println(object.getKey());
object.close();
}
回答by HuTa
Check if HttpResponseHandler is closing connections from the pool. AmazonHttpClient has the paramter 'leaveHttpConnectionOpen' which indicates that the connection should be closed or not.
检查 HttpResponseHandler 是否正在关闭池中的连接。AmazonHttpClient 具有参数“leaveHttpConnectionOpen”，指示连接是否应该关闭。
