problems with static resources with HTTP/2 enabled /FISH-61

[eduarddrenth]

Description

---

I have a simple jsf 2 application with resources under javax.faces.resource. On and of there is a problem with downloading these, it either takes very long or fails, messing up the layout. Pressing F5 several times will finaly result in a correct page.

Expected Outcome

no problems with downloading static resources

Current Outcome

non responsive application with layout messed up

In the log I see:

• JSF1064: Unable to find or serve resource
• org.glassfish.grizzly.http.io.OutputBuffer.blockAfterWriteIfNeeded(OutputBuffer.java:1023)
• Unable to push resource identified by path
• org.glassfish.grizzly.nio.NIOConnection.assertOpen(NIOConnection.java:464)

Steps to reproduce (Only for bug reports)

Perhaps I'll spend time on reproducing, but I've just spent a few days already trying to solve this. My decision is to stick with 4.1.2.181 which is very stable.

Environment

• Payara Version: 5.181
• Edition: full
• JDK Version: 1.8.0_162-8u162-b12-0ubuntu0.16.04.2-b12
• Operating System: inux 4.4.0-119-generic
• Database: MySQL

[phillipross]

I've seen this too, and it's blocking us from moving to Payara5, but now that I've seen I'm not the only one, I think it may be worth the time to try to create a reproducer. In my case, the problem doesn't occur on every single request so it's a little difficult to reproduce.

[lprimak]

This looks like an issue with server push in HTTP/2. Can you confirm this?

[lprimak]

Also looks like a Grizzly issue

[phillipross]

I don't know about server push, but there's definitely lots of mention of grizzly in the stack traces. I'll try to setup another deployment of my project with payara 5.181 and gather some stack traces for reference.

[eduarddrenth]

I think I can manage to provide a reproducer of https://bitbucket.org/fryske-akademy/superlemma-list:

• payara preparation script (already there I think)
• mysql dump
• war (or just create that using mvn verify)
• it uses filerealm and needs a group Superlemma

Maybe tomorrow, but I think later....

[smillidge]

Try disabling http/2 on the listener and see if the issue goes away.

[lprimak]

Great idea

[eduarddrenth]

just tested, problem gone

[smillidge]

Ok looks like an issue with server push in http/2 as Lenny suggests.

[smillidge]

Sorry Lenny my phone called you Kenny.
We also have fixed some issues recently in Grizzly related to http/2 protocol handling so a simple test case would be really useful.

[phillipross]

Disabling HTTP/2 on the listeners worked for me too... I receive no errors after that.

[eduarddrenth]

shall I close this issue?

[lprimak]

Sorry Lenny my phone called you Kenny.
I get that a lot 🥇

[smillidge]

no leave open as it looks like a genuine issue

[eduarddrenth]

Reproducer

• install mysql
• install database from sup.sql.gz
• unzip payara
• put mysql-connector-java-6.0.6.jar.gz in glassfish/domains/domain1/lib
• download init_glassfish.sh and asadmin_commands.txt from https://bitbucket.org/fryske-akademy/superlemma-list
• adapt host and user in asadmin_commands.txt
• review and execute shellscript
• create a user in file realm with a group Superlemma
• clone https://bitbucket.org/fryske-akademy/superlemma-list
• mvn verify
• deploy war from superlemma-list-gui
• click around in the app

[MattGill98]

Hi @eduarddrenth,

The scope of that example is rather large, and I'm having trouble trying to narrow down the issue. Are you able to produce a concise reproducer?

Kind regards,

Matt

[eduarddrenth]

Hi, I am aware...I'll have to find some time somewhere....

[eduarddrenth]

Just tried fresh quarterly, layout isn't messed up anymore with http 2 on, but sometimes a page remains blank, after F5 I see the page. without http 2 all is well. See if I can give you a mvce anywhere soon

[eduarddrenth]

I see in the log:

Warning:   JSF1064: Unable to find or serve resource, layout/layout.js, from library, primefaces.
Warning:   java.io.IOException: Connection is closed
	at org.glassfish.grizzly.nio.NIOConnection.assertOpen(NIOConnection.java:440)

and later:

Severe:   Unable to push resource identified by path [/standertwurdlist-gui/faces/javax.faces.resource/jsf.js?ln=javax.faces]
Warning:   JSF1064: Unable to find or serve resource, hotkey/hotkey.js, from library, primefaces.
Severe:   Unable to push resource identified by path [/standertwurdlist-gui/faces/javax.faces.resource/datascroller/images/loading.gif?ln=primefaces&v=6.2.5]
Severe:   Session is closed
org.glassfish.grizzly.http2.Http2StreamException streamId=64 errorCode=REFUSED_STREAM: Session is closed
	at org.glassfish.grizzly.http2.Http2Session.openStream(Http2Session.java:1087)

[eduarddrenth]

maybe this helps: square/okhttp#2506

[eduarddrenth]

enabling http2, but disabling push solves the problem

[MattGill98]

Hi @eduarddrenth,

Thank you for your clarification. Unfortunately, I still need that reproducer. I can't seem to build the app you posted since it relies on some internal dependencies, and the setup is rather long which doesn't help me narrow down the issue. I've attempted my own reproducer here, but can't find the issue. Do you have a shorter, reproducible application, or a tweak to my current application to help me reproduce it?

Cheers,

Matt

[eduarddrenth]

I'll see what I can do, the app isn't realy that complicated, so I should be able to create a simple reproducer for you....may some time though.

[eduarddrenth]

For now we will not do http2, no functional need, apache not quite ready yet, a bit too early.

I'll still see if I can find some time to provide a reproducer.

[MattGill98]

Hi @eduarddrenth,

There's an issue raised with PrimeFaces that may be related. Are you able to reproduce the issue without using PrimeFaces?

Kind regards,

Matt

[eduarddrenth]

Thanks, that looks very similar indeed... From the discussion I understand there is no clarity whether it is a pf or a payara issue. I'm on holiday now. Reproducing without using pf is I suspect more of a hassle for me than providing a standalone reproducer....

[phillipross]

I've tried to reproduce the problem without primefaces but thus far have been unable to do so. I've not had time to make an isolated reproducer but would be willing to dedicate some time if this issue is going to get some attention.

[lprimak]

Assigned PAYARA-2890 issue. This is reproducible

[awenger63]

Hi, here one complet Log. It' exist a lot of same error for each resources files images, css, js ... than it's not only on PF graphicimage component also in java.io
@phillipross Error of number of ressources appear after a lot of this first error. I thinks, it's a consequence of the number of http request failed and resend.
Hope this can help.

Unable to push resource identified by path [/MyApp/javax.faces.resource/myLogo.png]|#] org.glassfish.grizzly.http2.Http2StreamException streamId=506 errorCode=REFUSED_STREAM at org.glassfish.grizzly.http2.Http2Session.openStream(Http2Session.java:1090) at org.glassfish.grizzly.http2.Http2ServerFilter.doPush(Http2ServerFilter.java:1017) at org.glassfish.grizzly.http2.Http2ServerFilter.handleEvent(Http2ServerFilter.java:603) at org.glassfish.grizzly.filterchain.ExecutorResolver$6.execute(ExecutorResolver.java:70) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53) at org.glassfish.grizzly.filterchain.FilterChainContext.notifyDownstream(FilterChainContext.java:929) at org.glassfish.grizzly.filterchain.FilterChainContext.notifyDownstream(FilterChainContext.java:911) at org.apache.catalina.core.ApplicationPushBuilder.push(ApplicationPushBuilder.java:365) at com.sun.faces.context.ExternalContextImpl.pushIfPossibleAndNecessary(ExternalContextImpl.java:1187) at com.sun.faces.context.ExternalContextImpl.encodeResourceURL(ExternalContextImpl.java:644) at javax.faces.context.ExternalContextWrapper.encodeResourceURL(ExternalContextWrapper.java:140) at org.primefaces.component.graphicimage.GraphicImageRenderer.getImageSrc(GraphicImageRenderer.java:71) at org.primefaces.component.graphicimage.GraphicImageRenderer.encodeEnd(GraphicImageRenderer.java:41) at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:595) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:286) at com.sun.faces.renderkit.html_basic.OutputLinkRenderer.encodeChildren(OutputLinkRenderer.java:100) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:566) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:278) at com.sun.faces.renderkit.html_basic.GroupRenderer.encodeChildren(GroupRenderer.java:90) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:566) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1647) at javax.faces.render.Renderer.encodeChildren(Renderer.java:152) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:566) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1647) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1650) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1650) at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:468) at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:170) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:132) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:102) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:76) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:199) at javax.faces.webapp.FacesServlet.executeLifecyle(FacesServlet.java:708) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:451) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1628) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:258) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:755) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:575) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:371) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53) at org.glassfish.grizzly.http2.Http2Session.sendMessageUpstream(Http2Session.java:1282) at org.glassfish.grizzly.http2.Http2Session.sendMessageUpstream(Http2Session.java:1258) at org.glassfish.grizzly.http2.Http2ServerFilter$2.run(Http2ServerFilter.java:1056) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) at java.lang.Thread.run(Thread.java:748)

[eduarddrenth]

On 5.194 I don't see

java.io.IOException: Write timeout exceeded when trying to flush the data
        at org.glassfish.grizzly.http.io.OutputBuffer.blockAfterWriteIfNeeded(OutputBuffer.java:999)

anymore, but though http2 is enabled I see http1.1 in FF 70:

[eduarddrenth]

chromium shows http1.1 as well

[rdebusscher]

Tracked now as CUSTCOM-22

[Budokai100]

The problem persists in 5.194:

[2019-12-31T10:16:16.236-0500] [Payara 5.194] [WARNING] [jsf.application.resource.unable_to_serve_from_library] [javax.enterprise.resource.webcontainer.jsf.application] [tid: _ThreadID=1186 _ThreadName=http-thread-pool::http-listener-2(24)] [timeMillis: 1577805376236] [levelValue: 900] [[
JSF1064: Unable to find or serve resource, fonts/rio.eot, from library, primefaces-rio;JSESSIONID=c876a861b38592128c8d2284bc25.]]

the server stops responding after a certain number of requests.

[oromancer]

Hello,

I had this problem too 5.192 + SSL + HTTP/2 with same error.

"The default is true" has confused me.

So I enabled "HTTP/2 Push Enabled" and problem dissappeared.
Works without problems on FireFox 68.0, Opera 64.0.3417.92.

[eduarddrenth]

Just tested again, with and without push, on 5.194 this: #2625 (comment) is still valid

[oromancer]

Hello,

I had this problem too 5.192 + SSL + HTTP/2 with same error.

"The default is true" has confused me.

So I enabled "HTTP/2 Push Enabled" and problem dissappeared.
Works without problems on FireFox 68.0, Opera 64.0.3417.92.

Sorry for that, the problem has reappeared today.

[smillidge]

I've bumped the priority on this.

[AndrewG10i]

Also getting this issue on 5.194 + SSL + HTTP/2. Disabling HTTP/2 for http-listener-2 (https) resolves the issue.
For us it is not so critical as usually we use nginx as reverse proxy in front of Payara, reporting about this issue for statistics purposes only. Thank you!

[james16694]

Moving from 4 to 5.194 build 327 + SSL + HTTP/2 we sometimes see HTML/images loading but not CSS/JS. We also have 2 websites on 5.192 + SSL + HTTP/2 which sometimes display a white screen / or partial loading as 5.194. Both seem related to this issue. We have now disabled HTTP/2 so far all good. We have also seen a class cast exception:

[2020-02-29T23:15:53.175+0000] [Payara 5.194] [WARNING] [] [org.glassfish.grizzly.filterchain.DefaultFilterChain] [tid: _ThreadID=47 _ThreadName=http-thread-pool::http-listener-2(2)] [timeMillis: 1583018153175] [levelValue: 900] [[
GRIZZLY0013: Exception during FilterChain execution
java.lang.ClassCastException: org.glassfish.grizzly.http2.Http2Request cannot be cast to org.glassfish.grizzly.http.HttpResponsePacket
at org.glassfish.grizzly.http2.Http2ServerFilter.processOutgoingHttpHeader(Http2ServerFilter.java:960)
at org.glassfish.grizzly.http2.Http2BaseFilter.handleWrite(Http2BaseFilter.java:729)
at org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:87)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:867)
at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:835)
at org.glassfish.grizzly.http.io.OutputBuffer.flushBuffer(OutputBuffer.java:1068)
at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:698)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:336)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:323)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:154)
at org.apache.catalina.servlets.DefaultServlet.copy(DefaultServlet.java:2139)
at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:1025)
at org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:508)

Thanks

[MattGill98]

Hi all,

Thank you all for your input on this issue. This has now spread into several issues, which the work on CUSTCOM-22 has gone towards resolving. This comment is to provide my summary of each of the issues as far as I understand them.

For each of the issues I've been using the reproducer built from https://bitbucket.org/luminositylabs/payara-2625.git.

Issue 1

Since these problems are all caused by HTTP/2 push, we advised that turning off HTTP/2 push would fix the issue. Despite this, the following errors could still be seen in the logs:

 Unable to push resource identified by path [/javax.faces.resource/css/styles.css.action?ln=layout]]]

org.glassfish.grizzly.http2.Http2StreamException streamId=204 errorCode=REFUSED_STREAM
        at org.glassfish.grizzly.http2.Http2Session.openStream(Http2Session.java:1090)
        at org.glassfish.grizzly.http2.Http2ServerFilter.doPush(Http2ServerFilter.java:1017)
        at org.glassfish.grizzly.http2.Http2ServerFilter.handleEvent(Http2ServerFilter.java:603)

This shows that HTTP/2 push is still, in some capacity, being used. CUSTCOM-22 has now been tagged and will be closed with the linked PR above. This fixes the help text of the admin console, and prevents these errors appearing when HTTP/2 push is disabled, therefore meaning that disabling HTTP/2 push should prevent these errors entirely.

Issue 2

The second issue is that when reloading the reproducer page repeatedly with push enabled, the following would occasionally happen:

A resource intended to be pushed is instead only partially pushed. A thread dump reveals that the server thinks it has finished processing the resource, but the browser is waiting on more. This will need a more complex solution, so CUSTCOM-277 has been created to track it. This may be related to the errors shown above, but that's not a certainty as sometimes this happens without the errors. I suspect that the errors may be part of the previous connection that couldn't be completed due to a subsequent page reload.

Issue 3

The third issue is the above error itself. As stated briefly above, this could not be a problem at all if the errors are due to the previous connection being cut by a page reload (essentially the HTTP/2 push version of https://stackoverflow.com/questions/642061/error-commiting-response-java-io-ioexception-broken-pipe-at-sun-nio-ch-filedisp). Once CUSTCOM-277 has been resolved, we can then begin to investigate this properly.

Thank you all for your patience with this issue. If the first issue I outlined is solved by the linked PR, I'll close this issue and reopen issues outlining the remaining HTTP/2 push problems.

[rdebusscher]

CUSTCOM-22 is part of release 5.2020.2. More work around HTTP/2 and HTTP Push is planned for next release.

[AlanRoth]

Hi, the issue seems to have been resolved in release 5.2020.2 and 5.20. I will be closing this issue, however feel free to raise a new issue if any bugs are spotted in the latest release. Thank you.

[eduarddrenth]

Just tested, I can confirm http2 is active and application runs fine. Thanks!

[lprimak]

Congratulations everyone! This one is near and dear to my heart.

[StevenHachel]

Sorry, but under Payara 5.2022.5 Community I still have the problem as soon as I switch to https and HTTP/2 (not push) is activated. After a short time, resources are simply not reloaded and the display of the pages is incorrect and the behavior is incorrect because the Javascripts sometimes do not work properly.
I have a Primefaces Enterprise application running that doesn't cause any problems under http and Http/2.
Switching to Payara 6 is currently not possible.