使用 REST 模板 Java Spring MVC 从服务器下载大文件
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/32988370/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Download Large file from server using REST template Java Spring MVC
提问by arpit joshi
I have a REST service which sends me a large ISO file ,there are no issues in the REST service . Now I have written a Web application which calls the rest service to get the file ,on the client(web app) side I receive a Out Of memory Exception.Below is my code
我有一个 REST 服务，它向我发送一个大的 ISO 文件，REST 服务中没有问题。现在我编写了一个 Web 应用程序，它调用其余服务来获取文件，在客户端（Web 应用程序）端我收到一个内存不足异常。下面是我的代码
HttpHeaders headers = new HttpHeaders();//1 Line
headers.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM));//2 Line
headers.set("Content-Type","application/json");//3 Line
headers.set("Cookie", "session=abc");//4 Line
HttpEntity statusEntity=new HttpEntity(headers);//5 Line
String uri_status=new String("http://"+ip+":8080/pcap/file?fileName={name}");//6 Line
ResponseEntity<byte[]>resp_status=rt.exchange(uri_status, HttpMethod.GET, statusEntity, byte[].class,"File5.iso");//7 Line
I receive out of memory exception at 7 line ,I guess i will have to buffer and get in parts ,but dont know how can i get this file from the server ,the size of the file is around 500 to 700 MB . Can anyone please assist .
我在第 7 行收到内存不足异常，我想我将不得不缓冲并获取部分内容，但不知道如何从服务器获取此文件，文件大小约为 500 到 700 MB。任何人都可以帮忙。
Exception Stack:
异常堆栈：
org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
root cause
java.lang.OutOfMemoryError: Java heap space
java.util.Arrays.copyOf(Arrays.java:3236)
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113)
org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164)
org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:58)
org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:1)
org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:153)
org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:81)
org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:627)
org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:1)
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:454)
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:409)
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:385)
com.pcap.webapp.HomeController.getPcapFile(HomeController.java:186)
My Server Side REST Service Code which is working fine is
我工作正常的服务器端 REST 服务代码是
@RequestMapping(value = URIConstansts.GET_FILE, produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE}, method = RequestMethod.GET)
public void getFile(@RequestParam(value="fileName", required=false) String fileName,HttpServletRequest request,HttpServletResponse response) throws IOException{
byte[] reportBytes = null;
File result=new File("/home/arpit/Documents/PCAP/dummyPath/"+fileName);
if(result.exists()){
InputStream inputStream = new FileInputStream("/home/arpit/Documents/PCAP/dummyPath/"+fileName);
String type=result.toURL().openConnection().guessContentTypeFromName(fileName);
response.setHeader("Content-Disposition", "attachment; filename=" + fileName);
response.setHeader("Content-Type",type);
reportBytes=new byte[100];//New change
OutputStream os=response.getOutputStream();//New change
int read=0;
while((read=inputStream.read(reportBytes))!=-1){
os.write(reportBytes,0,read);
}
os.flush();
os.close();
}
采纳答案by bernie
Here is how I do it. Based on hints from this Spring Jira issue.
这是我如何做到的。基于这个Spring Jira 问题的提示。
RestTemplate restTemplate // = ...;
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("some/path");
Files.copy(response.getBody(), path);
return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);
From the aforementioned Jira issue:
从上述 Jira 问题：
Note that you cannot simply return the InputStream from the extractor, because by the time the execute method returns, the underlying connection and stream are already closed.
请注意，您不能简单地从提取器返回 InputStream，因为到 execute 方法返回时，底层连接和流已经关闭。
Update for Spring 5
Spring 5 更新
Spring 5 introduced the WebClientclass which allows asynchronous (e.g. non-blocking) http requests. From the doc:
Spring 5 引入了WebClient允许异步（例如非阻塞）http 请求的类。从文档：
By comparison to the RestTemplate, the WebClient is:
- non-blocking, reactive, and supports higher concurrency with less hardware resources.
- provides a functional API that takes advantage of Java 8 lambdas.
- supports both synchronous and asynchronous scenarios.
- supports streaming up or down from a server.
与 RestTemplate 相比，WebClient 是：
- 非阻塞，反应性，并以更少的硬件资源支持更高的并发性。
- 提供了一个利用 Java 8 lambdas 的函数式 API。
- 支持同步和异步场景。
- 支持从服务器向上或向下流式传输。
回答by Andrea Girardi
This prevents loading the entire request into memory.
这可以防止将整个请求加载到内存中。
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
RestTemplate rest = new RestTemplate(requestFactory);
For java.lang.OutOfMemoryError: Java heap space can be solved adding more memory to the JVM:
对于 java.lang.OutOfMemoryError：Java 堆空间可以通过向 JVM 添加更多内存来解决：
-Xmxn Specifies the maximum size, in bytes, of the memory allocation pool. This value must a multiple of 1024 greater than 2 MB. Append the letter k or K to indicate kilobytes, or m or M to indicate megabytes. The default value is chosen at runtime based on system configuration.
For server deployments, -Xms and -Xmx are often set to the same value. See Garbage Collector Ergonomics at http://docs.oracle.com/javase/7/docs/technotes/guides/vm/gc-ergonomics.html
Examples:
-Xmx83886080
-Xmx81920k
-Xmx80m
-Xmxn 指定内存分配池的最大大小（以字节为单位）。该值必须是大于 2 MB 的 1024 的倍数。附加字母 k 或 K 以指示千字节，或附加字母 m 或 M 以指示兆字节。默认值是在运行时根据系统配置选择的。
对于服务器部署，-Xms 和 -Xmx 通常设置为相同的值。请参阅垃圾收集器人体工程学，网址为 http://docs.oracle.com/javase/7/docs/technotes/guides/vm/gc-ergonomics.html
例子：
-Xmx83886080
-Xmx81920k
-Xmx80m
Probably the problem you have is not strictly related to the request you are trying to execute (download large file) but the memory allocated for the process is not enough.
您遇到的问题可能与您尝试执行的请求（下载大文件）并不严格相关，但为该进程分配的内存不够。
回答by Donald
You should use multipart file attachment, so the file stream isn't load into memory. In this example, I use a rest service implemented with Apache CXF.
您应该使用多部分文件附件，因此文件流不会加载到内存中。在这个例子中，我使用了一个用 Apache CXF 实现的休息服务。
...
import org.apache.cxf.jaxrs.ext.multipart.Attachment;
...
@Override
@Path("/put")
@Consumes("multipart/form-data")
@Produces({ "application/json" })
@POST
public SyncResponseDTO put( List<Attachment> attachments) {
SyncResponseDTO response = new SyncResponseDTO();
try {
for (Attachment attr : attachments) {
log.debug("get input filestream: " + new Date());
InputStream is = attr.getDataHandler().getInputStream();
回答by Rajdeep
A better version of above correct answer could be the below code. This method will send download request to another application or service acting as actual source of truth for downloaded information.
上面正确答案的更好版本可能是下面的代码。此方法将向另一个应用程序或服务发送下载请求，该应用程序或服务充当下载信息的实际真实来源。
public void download(HttpServletRequest req, HttpServletResponse res, String url)
throws ResourceAccessException, GenericException {
try {
logger.info("url::" + url);
if (restTemplate == null)
logger.info("******* rest template is null***********************");
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<ResponseEntity<InputStream>> responseExtractor = response -> {
String contentDisposition = response.getHeaders().getFirst("Content-Disposition");
if (contentDisposition != null) {
// Temporary location for files that will be downloaded from micro service and
// act as final source of download to user
String filePath = "/home/devuser/fileupload/download_temp/" + contentDisposition.split("=")[1];
Path path = Paths.get(filePath);
Files.copy(response.getBody(), path, StandardCopyOption.REPLACE_EXISTING);
// Create a new input stream from temporary location and use it for downloading
InputStream inputStream = new FileInputStream(filePath);
String type = req.getServletContext().getMimeType(filePath);
res.setHeader("Content-Disposition", "attachment; filename=" + contentDisposition.split("=")[1]);
res.setHeader("Content-Type", type);
byte[] outputBytes = new byte[100];
OutputStream os = res.getOutputStream();
int read = 0;
while ((read = inputStream.read(outputBytes)) != -1) {
os.write(outputBytes, 0, read);
}
os.flush();
os.close();
inputStream.close();
}
return null;
};
restTemplate.execute(url, HttpMethod.GET, requestCallback, responseExtractor);
} catch (Exception e) {
logger.info(e.toString());
throw e;
}
}
回答by Krzysztof Skrzynecki
As @berniementioned you can use WebClient to achieve this:
正如@bernie提到的，您可以使用 WebClient 来实现这一点：
public void downloadFileUrl( HttpServletResponse response ) throws IOException {
WebClient webClient = WebClient.create();
// Request service to get file data
Flux<DataBuffer> fileDataStream = webClient.get()
.uri( this.fileUrl )
.accept( MediaType.APPLICATION_OCTET_STREAM )
.retrieve()
.bodyToFlux( DataBuffer.class );
// Streams the stream from response instead of loading it all in memory
DataBufferUtils.write( fileDataStream, response.getOutputStream() )
.map( DataBufferUtils::release )
.then()
.block();
}
You can still use WebClient even if you don't have Reactive Server stack - Rossen Stoyanchev (a member of Spring Framework team) explains it quite well in the Guide to "Reactive" for Spring MVC Developerspresentation. During this presentation, Rossen Stoyanchev mentioned that they thought about deprecating RestTemplate, but they have decided to postpone it after all, but it may still happen in the future!
即使您没有 Reactive Server 堆栈，您仍然可以使用 WebClient - Rossen Stoyanchev（Spring 框架团队的成员）在Spring MVC Developers演示文稿的“Reactive”指南中对其进行了很好的解释。在这次演讲中，Rossen Stoyanchev 提到他们考虑过弃用 RestTemplate，但他们毕竟决定推迟它，但它可能会在未来发生！
The main disadvantage of using WebClient so far it's a quite steep learning curve (reactive programming), but I think there is no way to avoid in the future, so better to take a look on it sooner than latter.
到目前为止，使用 WebClient 的主要缺点是一个非常陡峭的学习曲线（反应式编程），但我认为将来没有办法避免，所以最好早点看一看。
