Sending large payloads with RestTemplate results in an OutOfMemoryError [SPR-7909]

[spring-projects-issues]

Arjen Poutsma opened SPR-7909 and commented

When sending a large file with the RestTemplate (with a POST or PUT), the RestTemplate will throw an OutOfMemoryError:

com.cat.tcm.jmx.ManagementException: An unexpected error occurred performing the operation. 
at com.cat.tcm.jmx.JMXTaskTemplate.run(JMXTaskTemplate.java:144) 
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 
at java.lang.Thread.run(Thread.java:619) 
Caused by: java.lang.OutOfMemoryError: Java heap space 
at java.util.Arrays.copyOf(Arrays.java:2786) 
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) 
at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113) 
at org.springframework.http.converter.ResourceHttpMessageConverter.write(ResourceHttpMessageConverter.java:85) 
at org.springframework.http.converter.ResourceHttpMessageConverter.write(ResourceHttpMessageConverter.java:1) 
at org.springframework.http.converter.FormHttpMessageConverter.writePart(FormHttpMessageConverter.java:288) 
at org.springframework.http.converter.FormHttpMessageConverter.writeParts(FormHttpMessageConverter.java:252) 
at org.springframework.http.converter.FormHttpMessageConverter.writeMultipart(FormHttpMessageConverter.java:242) 
at org.springframework.http.converter.FormHttpMessageConverter.write(FormHttpMessageConverter.java:194) 
at org.springframework.http.converter.FormHttpMessageConverter.write(FormHttpMessageConverter.java:1) 
at org.springframework.web.client.RestTemplate$HttpEntityRequestCallback.doWithRequest(RestTemplate.java:588) 
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:436) 
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:401) 
at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:279) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309) 
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) 
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) 
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:50) 
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) 
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) 
at $Proxy17.postForObject(Unknown Source)

---

Issue Links:

• INT-2006 http:inbound-gateway/outbound-gateway content-length needs recalculation

Referenced from: commits d0d6a07, 1ed1c59

[spring-projects-issues]

Arjen Poutsma commented

This is due to the fact that we buffer all content before sending it over the wire, which is necessary for setting the correct Content-Length header.

[spring-projects-issues]

Arjen Poutsma commented

Fixed, by adding a 'bufferRequestBody' property to the SimpleClientHttpRequestFactory, which by default is true.

Setting this property to false will enable streaming mode. This will result in a ClientHttpRequest that either streams directly to the underlying HttpURLConnection (if the Content-Length is known in advance), or that will use "Chunked transfer encoding" (if the Content-Length is not known in advance). That is:

SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
RestTemplate rest = new RestTemplate(requestFactory);

These changes will be in 3.0.6, and in 3.1 M1. Please try a recent 3.1 M1 (as of tonight) to see if it works for you.

[spring-projects-issues]

Juergen Hoeller commented

I've removed this from the 3.0.6 backport list since it really seems to be a new 3.1 feature to me. Unless there is a strong need to have this in 3.0.x, I'd rather recommend an upgrade to 3.1 instead.

Juergen

[spring-projects-issues]

Serge P. Nekoval commented

Any chance to get it fixed for HttpComponentsClientHttpRequestFactory?

[ghost]

How to add this property with feign instead of restTemplate?

[SauriBabu]

@poutsma can you help me with below confusions.
How to set content-length ? using HttpHeaders ?
How much should be content-length ? Should it be equal to fileSize or byte size by which we want to buffer ?