Using AJP when downloading of large streaming data results in the errors "OutOfMemoryError: Java heap space" and "503 Service Temporarily Unavailable" #350
Comments
|
Try bumping up the heap size via jvm parameters in your domain or test with the payaradomain which has higher memory heapsize settings than the default. It would be helpful to know if this is reproducible when the memory settings are configured more generously than the fairly constrained defaults (512M) that ship with glassfish. Ideally, the JVM should not run out of memory. If you haven't already changed the default settings in the default domain, you can use these asadmin commands: asadmin delete-jvm-options -Xmx512m asadmin create-jvm-options -Xmx1g asadmin restart-domain domain1 |
|
Can you get a heap dump using -XX:+HeapDumpOnOutOfMemoryError http://www.oracle.com/technetwork/java/javase/clopts-139448.html#gbzrr then run through Eclipse Memory Analyzer to see what is holding the reference to all the memory. |
|
@smillidge thanks for your comment! I'm new to getting heap dumps and Eclipse Memory Analyzer but I stopped domain1, added I reproduced the OutOfMemoryError with this... ... and posted the heap dump here: http://dvn-vm1.hmdc.harvard.edu/tmp/payara/issues/350/java_pid23376.hprof This is with the following Apache config: This is the war file I'm using: https://github.com/IQSS/apachetest/releases/download/v0.1/apachetest-0.1.war I used the Eclipse Memory Analyzer wizard to create this screenshot of "Problem Suspect 1" for leaks, if it's helpful:
At the top it says this:
The "Shortest Paths To the Accumulation Point" looks like this:
I hope this helps! Please let me know if there is any more information I can provide. |
|
Thanks that's a lot to go on. It looks like the ajp buffer is not being flushed. |
…cified number bytes through a servlet output stream.
|
@smillidge sure thing. Not to distract you but @phillipross and I ran a different experiment with just a servlet you might want to check out, including another heap dump: IQSS/apachetest@488fe9e I'm not sure what it shows exactly but perhaps @phillipross can better explain. @smillidge I'm not sure how you've concluded that the ajp buffer isn't being flushed but I trust you. :) Are you saying this is something that could be fixed? |
|
The servlet test just shows that jersey handling of streaming is not necessarily the issue that triggers the problem. It occurs even when a simple servlet writing lots of data (with regular explicit flushes) to the servlet output stream. |
Perhaps https://java.net/jira/browse/GRIZZLY-1709 is unrelated. Payara 4.1.152.1 (the version I've been testing with) already includes Grizzly 2.3.19 and the fix version was 2.3.17. Sorry to clutter this issue with a false alarm. ( http://download.oracle.com/glassfish/4.1/nightly/glassfish-4.1-b14-06_22_2015.zip also includes Grizzly 2.3.19 . I just repeated the test with that "nightly" and got the same OutOfMemoryError.) So maybe this is a different bug in Grizzly that may not even have a bug number yet? |
|
Hi, I work with @pdurbin on the Dataverse project. I just want to add one (potentially) important bit of information: When there is no content-length header - on direct http connections, Grizzly switches into transfer-encoding=chunked, and starts delivering content, flushing after every write. |
|
Thanks sounds like key information for narrowing this down. I'll get a test case together and raise on the Grizzly JIRA while we work on this issue. |
|
Can you test the patch attached to https://java.net/jira/browse/GRIZZLY-1787 if it works we'll add to our patched projects repo. |
|
@smillidge thank you so much for creating that issue and thank you @rlubke for creating the patch! I've been googling but I'm not quite sure how to apply the patch. Is it simply a matter of copying the updated "grizzly-http-ajp-2.3.19.jar" file into payara41/glassfish/modules? |
|
You'll want to make a backup of glassfish-grizzly-extra-all.jar and then replace the org.glassfish.grizzly.http.ajp.AjpHttpRequest class from the extra-all.jar with the same class from the patch jar. |
|
Yep @rlubke is correct, that's the only way at the moment until we load the patched jar into https://github.com/payara/Payara_PatchedProjects/tree/master/org/glassfish/grizzly/grizzly-http-ajp with a specific patched rev number and mess with the build dependencies to pick it up. Which we can do if it fixes your issue. @rlubke do you know which Grizzly release this will make it into? |
|
The next release version is 2.3.22. I'll have to check with the team regarding a release timeline. |
|
Oops just saw the JIRA update 2.3.22 awesome. That might be out before our end of July release in which case we'll pick that up otherwise we'll do a patched Grizzly until then. |
|
The patch at https://java.net/jira/secure/attachment/55000/grizzly-http-ajp-2.3.19.jar from https://java.net/jira/browse/GRIZZLY-1787 works! Here's how I patched the jar (with double checks of the new size of Before putting the patched jar in place I repeated the Then I stopped Payara, put the patched Then I repeated the test. There was no OutOfMemoryError in Speaking of release timelines, does anyone know when a version of Grizzly that contains this patch would appear in Glassfish? I'll certainly report this recent activity on the thread I started (but I'll probably wait until there is a commit I can link to): https://java.net/projects/glassfish/lists/users/archive/2015-06/message/13 |
|
Thanks we'll look to incorporate the patch into the build asap. Can't speak for GlassFish but I can imagine it will be when they up-rev Grizzly in their nightly build to 2.3.22. Nightly GlassFish currently has 2.3.21 |
smillidge
added
reproducible
and removed
PR: TESTS REQUIRED
|
Glad to hear it works. I'll follow up here once I have an idea on the release timeline. |
|
@rlubke thanks! And I see you already committed the fix at https://java.net/projects/grizzly/sources/git/diff/modules/http-ajp/src/main/java/org/glassfish/grizzly/http/ajp/AjpHttpRequest.java?rev1=e9ea48db1b28405885d666475ba0d0c2f1abdcc4&rev2=102c432aa2004c581db1c297e94b0eb90dd74122 Awesome. @rlubke Do you have any suggestions on how to patch Glassfish 4.1? Glassfish 4.1 seems to have Grizzly 2.3.15-1, which is much older than the 2.3.19 version we've been testing with here. I'd be nervous about taking a class file from 2.3.19 and sticking it in jar file for 2.3.15-1. |
|
Nothing has changed between 2.3.15-1 (July 2014) and 2.3.19 so I imagine it is pretty safe to do. |
|
@pdurbin Yep, @smillidge beat me to it. Very little change in this area. |
|
To be clear, are we saying it safe to drop the updated "AjpHttpRequest.class" file from https://java.net/jira/secure/attachment/55000/grizzly-http-ajp-2.3.19.jar into the "glassfish-grizzly-extra-all.jar" file that's in Glassfish 4.1 (which was built from 2.3.15-1)? |
|
@pdurbin Correct. |
|
Yep, the patched "AjpHttpRequest.class" seems to work fine even in Glassfish 4.1: IQSS/dataverse#2180 (comment) . Thanks! |
PAYARA-322 fixes #350 upgrades Grizzly to 2.3.22
|
Just a quick follow up. Grizzly 2.3.22, which includes the fix for this issue, was recently released. |
|
Thanks @rlubke. I had some problems incorporating Grizzly 2.3.22 as I had to disable the Http2 addon to get it to work. I also had to add the dependencies grizzly-http2 and grizzly-npn-api. Would be useful to know how to enable Http2 correctly with the latest Grizzly on GlassFish. |
|
We haven't done the integration work for HTTP/2+Glassfish yet. Would you mind opening an issue to track that work? |
|
On Grizzly JIRA? |
|
Yeah, that'd be fine. |
|
https://java.net/jira/browse/GRIZZLY-1803 raised on Grizzly for HTTP2 support in GlassFish |
…lease Payara enterprise 5.27.0 release into master
Summary
I'm using Payara 4.1.152.1 behind Apache 2.2 (specifically, the version that ships with CentOS 6:
httpd-2.2.15-39.el6.centos.x86_64) and I'm gettingjava.lang.OutOfMemoryError: Java heap spaceand "503 Service Temporarily Unavailable" when I try to use AJP (mod_proxy_ajp) when downloading a gigabyte from a simple app that allows an arbitrary number of bytes to be streamed. I don't see the problem when I don't use AJP.How to reproduce
asadmin start-domain domain1asadmin create-network-listener --protocol http-listener-1 --listenerport 8009 --jkenabled true jk-connectorExpected behavior: The file is downloaded without incident
Actual behavior: Payara logs
java.lang.OutOfMemoryError: Java heap spaceand the client reports "503 Service Temporarily Unavailable".Without AJP everything works
Below is the output of
curlwhen AJP is not used. Everything works fine. No 503 errors. The Apache config looks like this:The most important output below is the line indicating
200 OK. The one gigabyte file was downloaded without indicent:Using AJP results in OutOfMemoryError and 503 errors
In order to use AJP we first enable 8009 as the listener port:
asadmin create-network-listener --protocol http-listener-1 --listenerport 8009 --jkenabled true jk-connectorThen we modify the Apache config to use
mod_proxy_ajprather thanmod_proxy_http:Then we repeat the curl command that worked fine when we were not using AJP. Notice that we see "503 Service Temporarily Unavailable":
Here is the
OutOfMemoryErrorerror we see in /home/payara/payara41/glassfish/domains/domain1/logs/server.logThe text was updated successfully, but these errors were encountered: