Jersey Entity Filter Threads Racing issue leads to Corrupted Entity Graph and Object Graph

[shuangxipei]

We are using Jersey version 2.27
The Jersey configuration has the SelectableEntityFilteringFeature configured.
The server environment is Felix server with Jetty.
When we send two concurrent same Web Service API calls to the server, the entity in the response got corrupted, i.e., some attributes in the entity got lost.
This is intermittent, but if it happens, all subsequent calls to that corrupted web service API will got back the same corrupted entity with same attributes missed.

Correct response entity example

[
{
"jobInstanceId": "1",
"jobId": "1",
"startTime": "2019-06-26T01:52:54",
"instanceState": {
"state": "FINISHED",
"detail": "Data discovery job finished successfully",
"reportedTime": "2019-06-26T01:53:04",
"stateSeq": 15
},
"schedule": "NOW",
},
{
"jobInstanceId": "2",
"jobId": "2",
"startTime": "2019-06-26T01:55:24",
"instanceState": {
"state": "FINISHED",
"detail": "Data discovery job finished successfully",
"reportedTime": "2019-06-26T01:55:38",
"stateSeq": 30
},
"schedule": "NOW",
},
{
"jobInstanceId": "3",
"jobId": "3",
"startTime": "2019-06-26T02:06:52",
"instanceState": {
"state": "FINISHED",
"detail": "Data discovery job finished successfully",
"reportedTime": "2019-06-26T02:07:06",
"stateSeq": 45
},
"schedule": "NOW",
}
]

Corrupted response entity example

[
{
"jobInstanceId": "1"
},
{
"jobInstanceId": "2"
},
{
"jobInstanceId": "3"
}
]

How to reproduce:

1. Restart the JVM, sent two concurrent Web Service calls for the same URL (i.e, invoke the same web service at the same time).
   2) check if the response entity corrupted or not
   3) if not corrupted, repeat from step 1). You may need many times retries.

Observed Jersey Exception stack trace (occurred in both Moxy provider and Jackson provide.)

With Moxy Provider

|Caused by: 
|java.util.ConcurrentModificationException
|	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
|	at java.util.HashMap$KeyIterator.next(HashMap.java:1466)
|	at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
|	at org.glassfish.jersey.message.filtering.EntityGraphImpl.getFilteringScopes(EntityGraphImpl.java:168)
|	at org.glassfish.jersey.message.filtering.EntityGraphImpl.createFilteringScope(EntityGraphImpl.java:233)
|	at org.glassfish.jersey.message.filtering.EntityGraphImpl.addField(EntityGraphImpl.java:102)
|	at org.glassfish.jersey.message.filtering.EntityGraphImpl.addField(EntityGraphImpl.java:62)
|	at org.glassfish.jersey.message.filtering.spi.AbstractEntityProcessor.addFilteringScopes(AbstractEntityProcessor.java:134)
|	at org.glassfish.jersey.message.filtering.SelectableEntityProcessor.process(SelectableEntityProcessor.java:70)
|	at org.glassfish.jersey.message.filtering.spi.AbstractEntityProcessor.process(AbstractEntityProcessor.java:89)
|	at org.glassfish.jersey.message.filtering.EntityInspectorImpl.inspectEntityProperties(EntityInspectorImpl.java:173)
|	at org.glassfish.jersey.message.filtering.EntityInspectorImpl.inspect(EntityInspectorImpl.java:104)
|	at org.glassfish.jersey.message.filtering.spi.AbstractObjectProvider.getFilteringObject(AbstractObjectProvider.java:90)
|	at org.glassfish.jersey.message.filtering.spi.AbstractObjectProvider.getFilteringObject(AbstractObjectProvider.java:84)
|	at org.glassfish.jersey.moxy.json.internal.FilteringMoxyJsonProvider.preWriteTo(FilteringMoxyJsonProvider.java:80)
|	at org.eclipse.persistence.jaxb.rs.MOXyJsonProvider.writeTo(MOXyJsonProvider.java:958)
|	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266)
|	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251)
|	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
|	at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109)
|	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
|	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
|	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
|	at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135)
|	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662)
|	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395)
|	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385)
|	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280)
|	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
|	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
|	at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
|	at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
|	at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
|	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
|	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
|	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
|	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
|	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
|	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
|	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
|	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
|	at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:123)
|	at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:86)
|	at org.apache.felix.http.base.internal.dispatch.Dispatcher$1.doFilter(Dispatcher.java:146)
|	at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager.invokePreprocessors(WhiteboardManager.java:988)
|	at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:91)
|	at org.apache.felix.http.base.internal.dispatch.DispatcherServlet.service(DispatcherServlet.java:49)
|	at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
|	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:876)
|	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
|	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
|	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
|	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
|	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
|	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
|	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
|	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
|	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
|	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
|	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
|	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
|	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
|	at org.eclipse.jetty.server.Server.handle(Server.java:505)
|	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
|	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
|	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
|	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
|	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
|	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
|	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
|	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
|	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
|	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
|	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
|	at java.lang.Thread.run(Thread.java:748)

Similar stack trace for the Jackson Provider

Caused by:
|java.util.ConcurrentModificationException
| at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
| at java.util.HashMap$EntryIterator.next(HashMap.java:1476)
| at java.util.HashMap$EntryIterator.next(HashMap.java:1474)
| at org.glassfish.jersey.internal.guava.AbstractMapBasedMultimap$AsMap$AsMapIterator.next(AbstractMapBasedMultimap.java:1348)
| at org.glassfish.jersey.internal.guava.AbstractMapBasedMultimap$AsMap$AsMapIterator.next(AbstractMapBasedMultimap.java:1336)
| at org.glassfish.jersey.internal.guava.TransformedIterator.next(TransformedIterator.java:45)
| at org.glassfish.jersey.internal.guava.AbstractMultimap.containsValue(AbstractMultimap.java:41)
| at org.glassfish.jersey.internal.guava.HashMultimap.containsValue(HashMultimap.java:44)
| at org.glassfish.jersey.message.filtering.EntityGraphImpl.presentInScopes(EntityGraphImpl.java:205)
| at org.glassfish.jersey.message.filtering.DefaultEntityProcessor.process(DefaultEntityProcessor.java:95)
| at org.glassfish.jersey.message.filtering.DefaultEntityProcessor.process(DefaultEntityProcessor.java:79)
| at org.glassfish.jersey.message.filtering.EntityInspectorImpl.inspectEntityProperties(EntityInspectorImpl.java:173)
| at org.glassfish.jersey.message.filtering.EntityInspectorImpl.inspect(EntityInspectorImpl.java:104)
| at org.glassfish.jersey.message.filtering.spi.AbstractObjectProvider.getFilteringObject(AbstractObjectProvider.java:90)
| at org.glassfish.jersey.message.filtering.spi.AbstractObjectProvider.getFilteringObject(AbstractObjectProvider.java:84)
| at org.glassfish.jersey.jackson.internal.FilteringJacksonJaxbJsonProvider.writeTo(FilteringJacksonJaxbJsonProvider.java:130)
| at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266)
| at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251)
| at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
| at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109)
| at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
| at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
| at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)

Possible Root Cause :

When two requests received by the Jetty server at the same time, two threads are dispatched to service the requests at the same time. The two threads are going to create the Entity Graph and Object Graph by both calling the following Methods in

org.glassfish.jersey.message.filtering.spi.AbstractObjectProvider.getFilteringObject(final Class<?> entityClass, final boolean forWriter, final Annotation... annotations)

The implementation in above method is not threads safe!

Once the corrupted filteringObject is saved to the Guava cache, it stays there, so all subsequent calls will get the same corrupted response entity

So the fix is how to make the buildup of Entity Graph and Object Graph to be thread save.

[jansupol]

Duplicates #4187