Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

io.helidon.webserver.HttpPipelineTest hangs periodically #3301

Closed
ljnelson opened this issue Aug 19, 2021 · 16 comments
Closed

io.helidon.webserver.HttpPipelineTest hangs periodically #3301

ljnelson opened this issue Aug 19, 2021 · 16 comments
Assignees
@spericas @barchetta
Labels
intermittent P3 testing

Comments

@ljnelson
Copy link
Member

Periodically the io.helidon.webserver.HttpPipelineTest test case hangs seemingly indefinitely. I wanted to capture this issue here. There is no output.
@romain-grecourt
Copy link
Contributor

Thread dump:

2021-08-19 20:49:00
Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0.12+8-LTS-237 mixed mode):

Threads class SMR info:
_java_thread_list=0x00007f6120001c60, length=17, elements={
0x00007f615c024000, 0x00007f615c658000, 0x00007f615c65c000, 0x00007f615c672000,
0x00007f615c674000, 0x00007f615c676000, 0x00007f615c678000, 0x00007f615c67a000,
0x00007f615c711800, 0x00007f615cb26800, 0x00007f615d3f9800, 0x00007f60f4107800,
0x00007f60f410c000, 0x00007f615d67e800, 0x00007f60e8006000, 0x00007f60dc039800,
0x00007f6120001000
}

"main" #1 prio=5 os_prio=0 cpu=3586.06ms elapsed=12839.15s tid=0x00007f615c024000 nid=0x6747 runnable  [0x00007f61636aa000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(java.base@11.0.12/Native Method)
	at java.net.SocketInputStream.socketRead(java.base@11.0.12/SocketInputStream.java:115)
	at java.net.SocketInputStream.read(java.base@11.0.12/SocketInputStream.java:168)
	at java.net.SocketInputStream.read(java.base@11.0.12/SocketInputStream.java:140)
	at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.12/StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.12/StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(java.base@11.0.12/StreamDecoder.java:178)
	- locked <0x00000000ffc62770> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(java.base@11.0.12/InputStreamReader.java:181)
	at java.io.BufferedReader.fill(java.base@11.0.12/BufferedReader.java:161)
	at java.io.BufferedReader.readLine(java.base@11.0.12/BufferedReader.java:326)
	- locked <0x00000000ffc62770> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(java.base@11.0.12/BufferedReader.java:392)
	at io.helidon.webserver.utils.SocketHttpClient.receive(SocketHttpClient.java:174)
	at io.helidon.webserver.HttpPipelineTest.testPipelining(HttpPipelineTest.java:102)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.12/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.12/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.12/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.12/Method.java:566)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$155/0x0000000100132040.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall$$Lambda$156/0x0000000100132440.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$$Lambda$282/0x0000000100184c40.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$526/0x00000001002c6c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:206)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:131)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$236/0x0000000100178c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$235/0x0000000100178840.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$234/0x0000000100178440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$240/0x0000000100179c40.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@11.0.12/ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$236/0x0000000100178c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$235/0x0000000100178840.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$234/0x0000000100178440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$240/0x0000000100179c40.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@11.0.12/ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$236/0x0000000100178c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$235/0x0000000100178840.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$234/0x0000000100178440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator$$Lambda$198/0x000000010013d040.accept(Unknown Source)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=3.24ms elapsed=12839.11s tid=0x00007f615c658000 nid=0x674e waiting on condition  [0x00007f613d3f6000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.12/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@11.0.12/Reference.java:241)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.12/Reference.java:213)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=3.44ms elapsed=12839.11s tid=0x00007f615c65c000 nid=0x674f in Object.wait()  [0x00007f613d2f5000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.12/Native Method)
	- waiting on <0x00000000f8180a90> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.12/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000000f8180a90> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.12/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.12/Finalizer.java:170)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.47ms elapsed=12839.10s tid=0x00007f615c672000 nid=0x6750 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=0.08ms elapsed=12839.10s tid=0x00007f615c674000 nid=0x6751 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=4598.36ms elapsed=12839.10s tid=0x00007f615c676000 nid=0x6752 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=2119.32ms elapsed=12839.10s tid=0x00007f615c678000 nid=0x6753 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #8 daemon prio=9 os_prio=0 cpu=0.13ms elapsed=12839.09s tid=0x00007f615c67a000 nid=0x6754 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #9 daemon prio=8 os_prio=0 cpu=14.63ms elapsed=12839.03s tid=0x00007f615c711800 nid=0x6756 in Object.wait()  [0x00007f613ca7a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.12/Native Method)
	- waiting on <0x00000000f81ba240> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.12/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000000f81ba240> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@11.0.12/CleanerImpl.java:148)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)
	at jdk.internal.misc.InnocuousThread.run(java.base@11.0.12/InnocuousThread.java:134)

"surefire-forkedjvm-command-thread" #11 daemon prio=5 os_prio=0 cpu=132.67ms elapsed=12838.69s tid=0x00007f615cb26800 nid=0x6757 runnable  [0x00007f613c142000]
   java.lang.Thread.State: RUNNABLE
	at java.io.FileInputStream.readBytes(java.base@11.0.12/Native Method)
	at java.io.FileInputStream.read(java.base@11.0.12/FileInputStream.java:279)
	at java.io.BufferedInputStream.read1(java.base@11.0.12/BufferedInputStream.java:290)
	at java.io.BufferedInputStream.read(java.base@11.0.12/BufferedInputStream.java:351)
	- locked <0x00000000f81017e8> (a java.io.BufferedInputStream)
	at java.io.BufferedInputStream.fill(java.base@11.0.12/BufferedInputStream.java:252)
	at java.io.BufferedInputStream.read1(java.base@11.0.12/BufferedInputStream.java:292)
	at java.io.BufferedInputStream.read(java.base@11.0.12/BufferedInputStream.java:351)
	- locked <0x00000000f81017c0> (a java.io.BufferedInputStream)
	at org.apache.maven.surefire.api.util.internal.Channels$3.readImpl(Channels.java:214)
	at org.apache.maven.surefire.api.util.internal.AbstractNoninterruptibleReadableChannel.read(AbstractNoninterruptibleReadableChannel.java:54)
	at org.apache.maven.surefire.booter.spi.LegacyMasterProcessChannelDecoder.decode(LegacyMasterProcessChannelDecoder.java:80)
	at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:343)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"helidon-client-0" #17 daemon prio=5 os_prio=0 cpu=83.44ms elapsed=12836.05s tid=0x00007f615d3f9800 nid=0x6763 runnable  [0x00007f610195c000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
	- locked <0x00000000fed066b8> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000fed06660> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"pool-2-thread-1" #18 prio=5 os_prio=0 cpu=8.49ms elapsed=12835.94s tid=0x00007f60f4107800 nid=0x6764 waiting on condition  [0x00007f610185b000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)
	- parking to wait for  <0x00000000fed06a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.12/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.12/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.12/ScheduledThreadPoolExecutor.java:1170)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.12/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.12/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.12/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.12/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"ForkJoinPool.commonPool-worker-3" #19 daemon prio=10 os_prio=0 cpu=8.09ms elapsed=12835.46s tid=0x00007f60f410c000 nid=0x6765 waiting on condition  [0x00007f610175a000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)
	- parking to wait for  <0x00000000f84fab20> (a java.util.concurrent.ForkJoinPool)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.12/LockSupport.java:194)
	at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.12/ForkJoinPool.java:1628)
	at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.12/ForkJoinWorkerThread.java:183)

"nioEventLoopGroup-20-1" #56 prio=10 os_prio=0 cpu=2.03ms elapsed=12832.70s tid=0x00007f615d67e800 nid=0x678c runnable  [0x00007f6102df8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
	- locked <0x00000000fed071b0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000fed07158> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"nioEventLoopGroup-21-1" #57 prio=10 os_prio=0 cpu=87.44ms elapsed=12832.69s tid=0x00007f60e8006000 nid=0x678d runnable  [0x00007f6101558000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
	- locked <0x00000000fed774c8> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000fed784f0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"pool-3-thread-1" #58 prio=5 os_prio=0 cpu=2.45ms elapsed=12832.54s tid=0x00007f60dc039800 nid=0x678e waiting on condition  [0x00007f61036fc000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)
	- parking to wait for  <0x00000000fea9db48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.12/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.12/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.12/ScheduledThreadPoolExecutor.java:1170)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.12/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.12/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.12/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.12/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"Attach Listener" #59 daemon prio=9 os_prio=0 cpu=0.35ms elapsed=0.10s tid=0x00007f6120001000 nid=0xbda waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=1504.57ms elapsed=12839.12s tid=0x00007f615c64f800 nid=0x674d runnable  

"GC Thread#0" os_prio=0 cpu=848.63ms elapsed=12839.15s tid=0x00007f615c039000 nid=0x6748 runnable  

"GC Thread#1" os_prio=0 cpu=846.44ms elapsed=12838.69s tid=0x00007f6124001000 nid=0x6758 runnable  

"G1 Main Marker" os_prio=0 cpu=1.72ms elapsed=12839.15s tid=0x00007f615c241000 nid=0x6749 runnable  

"G1 Conc#0" os_prio=0 cpu=29.59ms elapsed=12839.15s tid=0x00007f615c243000 nid=0x674a runnable  

"G1 Refine#0" os_prio=0 cpu=2.48ms elapsed=12839.15s tid=0x00007f615c4af000 nid=0x674b runnable  

"G1 Young RemSet Sampling" os_prio=0 cpu=1713.24ms elapsed=12839.15s tid=0x00007f615c4b0800 nid=0x674c runnable  
"VM Periodic Task Thread" os_prio=0 cpu=6121.39ms elapsed=12839.03s tid=0x00007f615c708800 nid=0x6755 waiting on condition  

JNI global refs: 16, weak refs: 0

@romain-grecourt
Copy link
Contributor

Intermittent issue reported for that same Test class: #2614 ; unsure if that's actually related.

@spericas
Copy link
Member

Intermittent issue reported for that same Test class: #2614 ; unsure if that's actually related.

That other test was removed AFAICT.

@spericas
Copy link
Member

spericas commented Aug 20, 2021 •
edited
Loading

Test is blocked waiting to receive response for get1. Can't quite understand the server's state from the thread dump. Are all threads there?

@spericas spericas self-assigned this Aug 20, 2021
@romain-grecourt
Copy link
Contributor

Yes, that's the whole output of jstack

@romain-grecourt
Copy link
Contributor

We might want to turn on max log verbosity in the webserver unit tests.

@spericas spericas mentioned this issue Aug 24, 2021
Merged
@spericas
Copy link
Member

Couldn't really tell the reason for the hang based on the thread dump. Adding more logging info to the test. See PR #3312.

@m0mus m0mus added the P3 label Aug 26, 2021
@spericas
Copy link
Member

There have been a few fixes recently related to hangs. This test has not been failing AFAICT. Will reopen if necessary.

@romain-grecourt
Copy link
Contributor

See in https://builds.helidon.io/188BEF7E61579798118FD5AC11034E91/logs/16

It's merge check that is merged with
0eef027

2021-10-19 23:00:56
Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0.12+8-LTS-237 mixed mode):

Threads class SMR info:
_java_thread_list=0x00007ff4c4002480, length=22, elements={
0x00007ff500024000, 0x00007ff500658000, 0x00007ff50065c000, 0x00007ff500672000,
0x00007ff500674000, 0x00007ff500676000, 0x00007ff500678000, 0x00007ff50067a000,
0x00007ff500719000, 0x00007ff500b3a000, 0x00007ff50145d800, 0x00007ff50129d800,
0x00007ff484002000, 0x00007ff5010b0000, 0x00007ff48c002800, 0x00007ff48c00b800,
0x00007ff45800d000, 0x00007ff460007000, 0x00007ff50103b000, 0x00007ff45800e800,
0x00007ff460044800, 0x00007ff4c4001800
}

"main" #1 prio=5 os_prio=0 cpu=5200.49ms elapsed=8690.06s tid=0x00007ff500024000 nid=0x59a8 runnable  [0x00007ff506a1f000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(java.base@11.0.12/Native Method)
	at java.net.SocketInputStream.socketRead(java.base@11.0.12/SocketInputStream.java:115)
	at java.net.SocketInputStream.read(java.base@11.0.12/SocketInputStream.java:168)
	at java.net.SocketInputStream.read(java.base@11.0.12/SocketInputStream.java:140)
	at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.12/StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.12/StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(java.base@11.0.12/StreamDecoder.java:178)
	- locked <0x00000000ff467bd8> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(java.base@11.0.12/InputStreamReader.java:181)
	at java.io.BufferedReader.fill(java.base@11.0.12/BufferedReader.java:161)
	at java.io.BufferedReader.readLine(java.base@11.0.12/BufferedReader.java:326)
	- locked <0x00000000ff467bd8> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(java.base@11.0.12/BufferedReader.java:392)
	at io.helidon.webserver.utils.SocketHttpClient.receive(SocketHttpClient.java:265)
	at io.helidon.webserver.HttpPipelineTest.testPipelining(HttpPipelineTest.java:110)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.12/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.12/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.12/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.12/Method.java:566)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$155/0x0000000100131c40.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall$$Lambda$156/0x0000000100132040.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$$Lambda$282/0x0000000100184c40.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$541/0x00000001002d9840.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:206)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:131)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$236/0x0000000100178c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$235/0x0000000100178840.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$234/0x0000000100178440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$240/0x0000000100179c40.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@11.0.12/ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$236/0x0000000100178c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$235/0x0000000100178840.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$234/0x0000000100178440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$240/0x0000000100179c40.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@11.0.12/ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$236/0x0000000100178c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$235/0x0000000100178840.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$234/0x0000000100178440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator$$Lambda$198/0x000000010013d040.accept(Unknown Source)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=2.53ms elapsed=8690.02s tid=0x00007ff500658000 nid=0x59af waiting on condition  [0x00007ff4e4856000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.12/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@11.0.12/Reference.java:241)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.12/Reference.java:213)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=2.63ms elapsed=8690.02s tid=0x00007ff50065c000 nid=0x59b0 in Object.wait()  [0x00007ff4e4755000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.12/Native Method)
	- waiting on <0x00000000f81a5ee0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.12/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000000f81a5ee0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.12/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.12/Finalizer.java:170)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.51ms elapsed=8690.01s tid=0x00007ff500672000 nid=0x59b1 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=0.09ms elapsed=8690.01s tid=0x00007ff500674000 nid=0x59b2 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=7424.96ms elapsed=8690.01s tid=0x00007ff500676000 nid=0x59b3 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=2812.84ms elapsed=8690.01s tid=0x00007ff500678000 nid=0x59b4 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #8 daemon prio=9 os_prio=0 cpu=0.10ms elapsed=8690.01s tid=0x00007ff50067a000 nid=0x59b5 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #9 daemon prio=8 os_prio=0 cpu=11.84ms elapsed=8689.95s tid=0x00007ff500719000 nid=0x59b7 in Object.wait()  [0x00007ff4b7dfb000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.12/Native Method)
	- waiting on <0x00000000f81be610> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.12/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000000f81be610> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@11.0.12/CleanerImpl.java:148)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)
	at jdk.internal.misc.InnocuousThread.run(java.base@11.0.12/InnocuousThread.java:134)

"surefire-forkedjvm-command-thread" #11 daemon prio=5 os_prio=0 cpu=99.39ms elapsed=8689.58s tid=0x00007ff500b3a000 nid=0x59b8 runnable  [0x00007ff4b74c3000]
   java.lang.Thread.State: RUNNABLE
	at java.io.FileInputStream.readBytes(java.base@11.0.12/Native Method)
	at java.io.FileInputStream.read(java.base@11.0.12/FileInputStream.java:279)
	at java.io.BufferedInputStream.read1(java.base@11.0.12/BufferedInputStream.java:290)
	at java.io.BufferedInputStream.read(java.base@11.0.12/BufferedInputStream.java:351)
	- locked <0x00000000f8260b40> (a java.io.BufferedInputStream)
	at java.io.BufferedInputStream.fill(java.base@11.0.12/BufferedInputStream.java:252)
	at java.io.BufferedInputStream.read1(java.base@11.0.12/BufferedInputStream.java:292)
	at java.io.BufferedInputStream.read(java.base@11.0.12/BufferedInputStream.java:351)
	- locked <0x00000000f8260b18> (a java.io.BufferedInputStream)
	at org.apache.maven.surefire.api.util.internal.Channels$3.readImpl(Channels.java:214)
	at org.apache.maven.surefire.api.util.internal.AbstractNoninterruptibleReadableChannel.read(AbstractNoninterruptibleReadableChannel.java:54)
	at org.apache.maven.surefire.booter.spi.LegacyMasterProcessChannelDecoder.decode(LegacyMasterProcessChannelDecoder.java:80)
	at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:343)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"helidon-client-0" #17 daemon prio=5 os_prio=0 cpu=464.42ms elapsed=8686.80s tid=0x00007ff50145d800 nid=0x59c0 runnable  [0x00007ff4b4db9000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
	- locked <0x00000000f9766ec0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000f9766e68> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"nioEventLoopGroup-12-1" #36 prio=10 os_prio=0 cpu=2.72ms elapsed=8683.09s tid=0x00007ff50129d800 nid=0x59d5 runnable  [0x00007ff4b6ac0000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
	- locked <0x00000000fc300508> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000fc3004b0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"nioEventLoopGroup-13-1" #37 prio=10 os_prio=0 cpu=48.55ms elapsed=8683.08s tid=0x00007ff484002000 nid=0x59d6 runnable  [0x00007ff4b4cb8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
	- locked <0x00000000fc3007e0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000fc300788> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"nioEventLoopGroup-14-1" #39 prio=10 os_prio=0 cpu=4.26ms elapsed=8682.98s tid=0x00007ff5010b0000 nid=0x59d8 runnable  [0x00007ff4b4eba000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
	- locked <0x00000000fc341930> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000fc3418d8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"nioEventLoopGroup-15-1" #40 prio=10 os_prio=0 cpu=24.82ms elapsed=8682.97s tid=0x00007ff48c002800 nid=0x59d9 runnable  [0x00007ff48bdfe000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
	- locked <0x00000000fc341c08> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000fc341bb0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"nioEventLoopGroup-15-2" #41 prio=10 os_prio=0 cpu=9.45ms elapsed=8682.93s tid=0x00007ff48c00b800 nid=0x59da runnable  [0x00007ff4b61bc000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
	- locked <0x00000000fc344b20> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000fc344970> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"pool-3-thread-1" #144 prio=5 os_prio=0 cpu=1.81ms elapsed=8680.40s tid=0x00007ff45800d000 nid=0x5a43 waiting on condition  [0x00007ff48b9fa000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)
	- parking to wait for  <0x00000000ffd224d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.12/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.12/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.12/LinkedBlockingQueue.java:433)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.12/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.12/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.12/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"pool-4-thread-1" #145 prio=5 os_prio=0 cpu=4.26ms elapsed=8680.34s tid=0x00007ff460007000 nid=0x5a44 waiting on condition  [0x00007ff48b5f6000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)
	- parking to wait for  <0x00000000ffeb49b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.12/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.12/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.12/LinkedBlockingQueue.java:433)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.12/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.12/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.12/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"nioEventLoopGroup-62-1" #148 prio=10 os_prio=0 cpu=2.18ms elapsed=8680.19s tid=0x00007ff50103b000 nid=0x5a47 runnable  [0x00007ff48b7f8000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
	- locked <0x00000000ff5efbb0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ff5ef988> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"nioEventLoopGroup-63-1" #149 prio=10 os_prio=0 cpu=79.86ms elapsed=8680.18s tid=0x00007ff45800e800 nid=0x5a48 runnable  [0x00007ff48b6f7000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.12/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.12/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.12/SelectorImpl.java:124)
	- locked <0x00000000ff5f38b8> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ff5f3690> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.12/SelectorImpl.java:141)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"pool-5-thread-1" #150 prio=5 os_prio=0 cpu=3.07ms elapsed=8680.09s tid=0x00007ff460044800 nid=0x5a49 waiting on condition  [0x00007ff48bcfd000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)
	- parking to wait for  <0x00000000ff5e6ce8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.12/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.12/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.12/ScheduledThreadPoolExecutor.java:1170)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.12/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.12/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.12/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.12/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.12/Thread.java:834)

"Attach Listener" #151 daemon prio=9 os_prio=0 cpu=0.32ms elapsed=0.10s tid=0x00007ff4c4001800 nid=0x721c waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=366.98ms elapsed=8690.03s tid=0x00007ff50064f800 nid=0x59ae runnable  

"GC Thread#0" os_prio=0 cpu=122.95ms elapsed=8690.06s tid=0x00007ff500039000 nid=0x59a9 runnable  

"GC Thread#1" os_prio=0 cpu=111.92ms elapsed=8689.57s tid=0x00007ff4c8001000 nid=0x59b9 runnable  

"G1 Main Marker" os_prio=0 cpu=1.31ms elapsed=8690.06s tid=0x00007ff500241000 nid=0x59aa runnable  

"G1 Conc#0" os_prio=0 cpu=33.95ms elapsed=8690.06s tid=0x00007ff500243000 nid=0x59ab runnable  

"G1 Refine#0" os_prio=0 cpu=5.93ms elapsed=8690.06s tid=0x00007ff5004af000 nid=0x59ac runnable  

"G1 Young RemSet Sampling" os_prio=0 cpu=1214.46ms elapsed=8690.06s tid=0x00007ff5004b0800 nid=0x59ad runnable  
"VM Periodic Task Thread" os_prio=0 cpu=4230.36ms elapsed=8689.95s tid=0x00007ff500710000 nid=0x59b6 waiting on condition  

JNI global refs: 16, weak refs: 0

@spericas
Copy link
Member

The log linked above (https://builds.helidon.io/188BEF7E61579798118FD5AC11034E91/logs/16) shows an earlier failure:

20:36:13,687 [ERROR] io.helidon.webserver.PrematureConnectionCutTest.cutConnectionBefore100Continue  Time elapsed: 0.069 s  <<< FAILURE!
java.lang.AssertionError: 
Thread Thread-5 is in invalid state.
Expected: is <TERMINATED>
     but: was <RUNNABLE>

It would be nice to determine if there's a pattern here that is causing the HttpPipelineTest to fail. Let's monitor that.

@romain-grecourt
Copy link
Contributor

See in https://builds.helidon.io/063A51158A3E709301AE66CFB618EFEF/logs/16

2021-12-17 18:22:34
Full thread dump OpenJDK 64-Bit Server VM (17+35-2724 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007f2f7c0014b0, length=23, elements={
0x00007f2fc0024e20, 0x00007f2fc05e7430, 0x00007f2fc05e8770, 0x00007f2fc05ef320,
0x00007f2fc05f05b0, 0x00007f2fc05f19a0, 0x00007f2fc05f3340, 0x00007f2fc05f48b0,
0x00007f2fc05f5ce0, 0x00007f2fc06252b0, 0x00007f2fc0628320, 0x00007f2fc09bcf60,
0x00007f2fc107b4e0, 0x00007f2fc10c6260, 0x00007f2f5802a9a0, 0x00007f2fc10cbed0,
0x00007f2f44000f90, 0x00007f2f44002000, 0x00007f2f100022d0, 0x00007f2fc0ab40e0,
0x00007f2f34007830, 0x00007f2f10047ab0, 0x00007f2f7c000a90
}

"main" #1 prio=5 os_prio=0 cpu=7512.22ms elapsed=25663.93s tid=0x00007f2fc0024e20 nid=0x5b1a runnable  [0x00007f2fc8c94000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.SocketDispatcher.read0(java.base@17/Native Method)
	at sun.nio.ch.SocketDispatcher.read(java.base@17/SocketDispatcher.java:47)
	at sun.nio.ch.NioSocketImpl.tryRead(java.base@17/NioSocketImpl.java:261)
	at sun.nio.ch.NioSocketImpl.implRead(java.base@17/NioSocketImpl.java:312)
	at sun.nio.ch.NioSocketImpl.read(java.base@17/NioSocketImpl.java:350)
	at sun.nio.ch.NioSocketImpl$1.read(java.base@17/NioSocketImpl.java:803)
	at java.net.Socket$SocketInputStream.read(java.base@17/Socket.java:966)
	at sun.nio.cs.StreamDecoder.readBytes(java.base@17/StreamDecoder.java:270)
	at sun.nio.cs.StreamDecoder.implRead(java.base@17/StreamDecoder.java:313)
	at sun.nio.cs.StreamDecoder.read(java.base@17/StreamDecoder.java:188)
	- locked <0x00000000fcf02138> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(java.base@17/InputStreamReader.java:177)
	at java.io.BufferedReader.fill(java.base@17/BufferedReader.java:162)
	at java.io.BufferedReader.readLine(java.base@17/BufferedReader.java:329)
	- locked <0x00000000fcf02138> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(java.base@17/BufferedReader.java:396)
	at io.helidon.webserver.utils.SocketHttpClient.receive(SocketHttpClient.java:265)
	at io.helidon.webserver.HttpPipelineTest.testPipelining(HttpPipelineTest.java:111)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17/NativeMethodAccessorImpl.java:77)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@17/Method.java:568)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$169/0x0000000800c89e38.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall$$Lambda$170/0x0000000800c8a258.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$$Lambda$281/0x0000000800cb9b30.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$297/0x0000000800cc2e10.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:206)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:131)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$231/0x0000000800cb4b50.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$230/0x0000000800cb4928.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$229/0x0000000800cb4500.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$235/0x0000000800cb5668.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@17/ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$231/0x0000000800cb4b50.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$230/0x0000000800cb4928.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$229/0x0000000800cb4500.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$235/0x0000000800cb5668.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@17/ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$231/0x0000000800cb4b50.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$230/0x0000000800cb4928.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$229/0x0000000800cb4500.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator$$Lambda$193/0x0000000800ca4ef8.accept(Unknown Source)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=4.88ms elapsed=25663.90s tid=0x00007f2fc05e7430 nid=0x5b21 waiting on condition  [0x00007f2fc4bfe000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@17/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@17/Reference.java:253)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@17/Reference.java:215)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=2.78ms elapsed=25663.90s tid=0x00007f2fc05e8770 nid=0x5b22 in Object.wait()  [0x00007f2fc4afd000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@17/Native Method)
	- waiting on <no object reference available>
	at java.lang.ref.ReferenceQueue.remove(java.base@17/ReferenceQueue.java:155)
	- locked <0x00000000f81592e8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@17/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@17/Finalizer.java:172)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.92ms elapsed=25663.90s tid=0x00007f2fc05ef320 nid=0x5b23 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=25.51ms elapsed=25663.90s tid=0x00007f2fc05f05b0 nid=0x5b24 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Deflation Thread" #6 daemon prio=9 os_prio=0 cpu=482.29ms elapsed=25663.90s tid=0x00007f2fc05f19a0 nid=0x5b25 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=21158.52ms elapsed=25663.90s tid=0x00007f2fc05f3340 nid=0x5b26 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #8 daemon prio=9 os_prio=0 cpu=4181.16ms elapsed=25663.90s tid=0x00007f2fc05f48b0 nid=0x5b27 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #9 daemon prio=9 os_prio=0 cpu=118.73ms elapsed=25663.90s tid=0x00007f2fc05f5ce0 nid=0x5b28 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Notification Thread" #10 daemon prio=9 os_prio=0 cpu=0.15ms elapsed=25663.89s tid=0x00007f2fc06252b0 nid=0x5b29 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #11 daemon prio=8 os_prio=0 cpu=24.53ms elapsed=25663.89s tid=0x00007f2fc0628320 nid=0x5b2b in Object.wait()  [0x00007f2fa9770000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@17/Native Method)
	- waiting on <no object reference available>
	at java.lang.ref.ReferenceQueue.remove(java.base@17/ReferenceQueue.java:155)
	- locked <0x00000000f8159568> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@17/CleanerImpl.java:140)
	at java.lang.Thread.run(java.base@17/Thread.java:833)
	at jdk.internal.misc.InnocuousThread.run(java.base@17/InnocuousThread.java:162)

"surefire-forkedjvm-command-thread" #13 daemon prio=5 os_prio=0 cpu=204.75ms elapsed=25663.60s tid=0x00007f2fc09bcf60 nid=0x5b2c runnable  [0x00007f2fa966f000]
   java.lang.Thread.State: RUNNABLE
	at java.io.FileInputStream.readBytes(java.base@17/Native Method)
	at java.io.FileInputStream.read(java.base@17/FileInputStream.java:276)
	at java.io.BufferedInputStream.read1(java.base@17/BufferedInputStream.java:282)
	at java.io.BufferedInputStream.read(java.base@17/BufferedInputStream.java:343)
	- locked <0x00000000f8159768> (a java.io.BufferedInputStream)
	at java.io.BufferedInputStream.fill(java.base@17/BufferedInputStream.java:244)
	at java.io.BufferedInputStream.read1(java.base@17/BufferedInputStream.java:284)
	at java.io.BufferedInputStream.read(java.base@17/BufferedInputStream.java:343)
	- locked <0x00000000f8159790> (a java.io.BufferedInputStream)
	at org.apache.maven.surefire.api.util.internal.Channels$3.readImpl(Channels.java:214)
	at org.apache.maven.surefire.api.util.internal.AbstractNoninterruptibleReadableChannel.read(AbstractNoninterruptibleReadableChannel.java:54)
	at org.apache.maven.surefire.booter.spi.LegacyMasterProcessChannelDecoder.decode(LegacyMasterProcessChannelDecoder.java:80)
	at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:343)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"nioEventLoopGroup-4-1" #17 prio=10 os_prio=0 cpu=27.53ms elapsed=25659.97s tid=0x00007f2fc107b4e0 nid=0x5b32 runnable  [0x00007f2fa8bfe000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@17/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17/SelectorImpl.java:129)
	- locked <0x00000000f83094a8> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000f8309458> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@17/SelectorImpl.java:146)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"helidon-client-0" #18 daemon prio=5 os_prio=0 cpu=4124.34ms elapsed=25659.84s tid=0x00007f2fc10c6260 nid=0x5b33 runnable  [0x00007f2fa8afd000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@17/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17/SelectorImpl.java:129)
	- locked <0x00000000f87a0f58> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000f879f1b8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@17/SelectorImpl.java:146)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"nioEventLoopGroup-5-1" #19 prio=10 os_prio=0 cpu=143.46ms elapsed=25659.77s tid=0x00007f2f5802a9a0 nid=0x5b34 runnable  [0x00007f2fa89fc000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@17/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17/SelectorImpl.java:129)
	- locked <0x00000000f8311db0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000f8311d60> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@17/SelectorImpl.java:146)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"nioEventLoopGroup-6-1" #21 prio=10 os_prio=0 cpu=4.59ms elapsed=25659.54s tid=0x00007f2fc10cbed0 nid=0x5b36 runnable  [0x00007f2fa88fb000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@17/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17/SelectorImpl.java:129)
	- locked <0x00000000f8748e98> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000f8748e48> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@17/SelectorImpl.java:146)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"nioEventLoopGroup-7-1" #22 prio=10 os_prio=0 cpu=24.02ms elapsed=25659.53s tid=0x00007f2f44000f90 nid=0x5b37 runnable  [0x00007f2fa87fa000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@17/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17/SelectorImpl.java:129)
	- locked <0x00000000f87822f8> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000f8782200> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@17/SelectorImpl.java:146)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"nioEventLoopGroup-7-2" #23 prio=10 os_prio=0 cpu=13.26ms elapsed=25659.50s tid=0x00007f2f44002000 nid=0x5b38 runnable  [0x00007f2fa86f9000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@17/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17/SelectorImpl.java:129)
	- locked <0x00000000f874ed38> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000f874ec40> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@17/SelectorImpl.java:146)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"pool-3-thread-1" #163 prio=5 os_prio=0 cpu=3.66ms elapsed=25624.32s tid=0x00007f2f100022d0 nid=0x5bdd waiting on condition  [0x00007f2f99dfc000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17/Native Method)
	- parking to wait for  <0x00000000fcf24b18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17/ForkJoinPool.java:3463)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17/ForkJoinPool.java:3434)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17/AbstractQueuedSynchronizer.java:1623)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17/ScheduledThreadPoolExecutor.java:1170)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"nioEventLoopGroup-88-1" #210 prio=10 os_prio=0 cpu=1.90ms elapsed=25622.46s tid=0x00007f2fc0ab40e0 nid=0x5c0e runnable  [0x00007f2f982fc000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@17/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17/SelectorImpl.java:129)
	- locked <0x00000000fcf24d50> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000fcf24d68> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@17/SelectorImpl.java:146)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"nioEventLoopGroup-89-1" #211 prio=10 os_prio=0 cpu=78.51ms elapsed=25622.46s tid=0x00007f2f34007830 nid=0x5c0f runnable  [0x00007f2fa84f7000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@17/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17/SelectorImpl.java:129)
	- locked <0x00000000fcf25000> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000fcf25018> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@17/SelectorImpl.java:146)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"pool-8-thread-1" #212 prio=5 os_prio=0 cpu=2.58ms elapsed=25622.35s tid=0x00007f2f10047ab0 nid=0x5c10 waiting on condition  [0x00007f2f99efd000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17/Native Method)
	- parking to wait for  <0x00000000fcf252c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17/ForkJoinPool.java:3463)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17/ForkJoinPool.java:3434)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17/AbstractQueuedSynchronizer.java:1623)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17/ScheduledThreadPoolExecutor.java:1170)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"Attach Listener" #213 daemon prio=9 os_prio=0 cpu=0.47ms elapsed=0.10s tid=0x00007f2f7c000a90 nid=0x2297 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=693.08ms elapsed=25663.91s tid=0x00007f2fc05e3560 nid=0x5b20 runnable  

"GC Thread#0" os_prio=0 cpu=704.26ms elapsed=25663.93s tid=0x00007f2fc0244800 nid=0x5b1b runnable  

"GC Thread#1" os_prio=0 cpu=702.76ms elapsed=25663.60s tid=0x00007f2f90004410 nid=0x5b2d runnable  

"G1 Main Marker" os_prio=0 cpu=24.19ms elapsed=25663.92s tid=0x00007f2fc0246dd0 nid=0x5b1c runnable  

"G1 Conc#0" os_prio=0 cpu=2434.61ms elapsed=25663.92s tid=0x00007f2fc0247d30 nid=0x5b1d runnable  

"G1 Refine#0" os_prio=0 cpu=3.52ms elapsed=25663.92s tid=0x00007f2fc04a6570 nid=0x5b1e runnable  

"G1 Refine#1" os_prio=0 cpu=2.27ms elapsed=25658.41s tid=0x00007f2f94000a90 nid=0x5b3f runnable  

"G1 Service" os_prio=0 cpu=3741.24ms elapsed=25663.92s tid=0x00007f2fc04a7460 nid=0x5b1f runnable  

"VM Periodic Task Thread" os_prio=0 cpu=9715.26ms elapsed=25663.89s tid=0x00007f2fc0626c00 nid=0x5b2a waiting on condition  

JNI global refs: 17, weak refs: 0

@romain-grecourt
Copy link
Contributor

see https://builds.helidon.io/34BDF890BF0963A0854040C1EC515747

2022-01-27 17:47:58
Full thread dump OpenJDK 64-Bit Server VM (17+35-2724 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007fb1900014b0, length=18, elements={
0x00007fb1d4024e20, 0x00007fb1d45e7430, 0x00007fb1d45e8770, 0x00007fb1d45ef320,
0x00007fb1d45f05b0, 0x00007fb1d45f19a0, 0x00007fb1d45f3340, 0x00007fb1d45f48b0,
0x00007fb1d45f5ce0, 0x00007fb1d46152d0, 0x00007fb1d4617f50, 0x00007fb1d4984dc0,
0x00007fb1d50ca150, 0x00007fb15c0ace90, 0x00007fb1d4f06520, 0x00007fb16400e530,
0x00007fb1600a9c60, 0x00007fb190000a90
}

"main" #1 prio=5 os_prio=0 cpu=7657.66ms elapsed=64404.27s tid=0x00007fb1d4024e20 nid=0x5be7 runnable  [0x00007fb1dd440000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.SocketDispatcher.read0(java.base@17/Native Method)
	at sun.nio.ch.SocketDispatcher.read(java.base@17/SocketDispatcher.java:47)
	at sun.nio.ch.NioSocketImpl.tryRead(java.base@17/NioSocketImpl.java:261)
	at sun.nio.ch.NioSocketImpl.implRead(java.base@17/NioSocketImpl.java:312)
	at sun.nio.ch.NioSocketImpl.read(java.base@17/NioSocketImpl.java:350)
	at sun.nio.ch.NioSocketImpl$1.read(java.base@17/NioSocketImpl.java:803)
	at java.net.Socket$SocketInputStream.read(java.base@17/Socket.java:966)
	at sun.nio.cs.StreamDecoder.readBytes(java.base@17/StreamDecoder.java:270)
	at sun.nio.cs.StreamDecoder.implRead(java.base@17/StreamDecoder.java:313)
	at sun.nio.cs.StreamDecoder.read(java.base@17/StreamDecoder.java:188)
	- locked <0x00000000ff9bcc20> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(java.base@17/InputStreamReader.java:177)
	at java.io.BufferedReader.fill(java.base@17/BufferedReader.java:162)
	at java.io.BufferedReader.readLine(java.base@17/BufferedReader.java:329)
	- locked <0x00000000ff9bcc20> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(java.base@17/BufferedReader.java:396)
	at io.helidon.webserver.utils.SocketHttpClient.receive(SocketHttpClient.java:265)
	at io.helidon.webserver.HttpPipelineTest.testPipelining(HttpPipelineTest.java:111)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17/NativeMethodAccessorImpl.java:77)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@17/Method.java:568)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$147/0x0000000800c3b5a8.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall$$Lambda$148/0x0000000800c3b9c8.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$$Lambda$281/0x0000000800cb96b8.apply(Unknown Source)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$297/0x0000000800cc2970.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:206)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:131)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$231/0x0000000800cb4458.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$230/0x0000000800cb4230.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$229/0x0000000800cb3e08.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$235/0x0000000800cb4f70.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@17/ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$231/0x0000000800cb4458.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$230/0x0000000800cb4230.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$229/0x0000000800cb3e08.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$235/0x0000000800cb4f70.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@17/ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$231/0x0000000800cb4458.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$230/0x0000000800cb4230.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$229/0x0000000800cb3e08.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator$$Lambda$193/0x0000000800ca4890.accept(Unknown Source)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=4.24ms elapsed=64404.24s tid=0x00007fb1d45e7430 nid=0x5bee waiting on condition  [0x00007fb1d92fc000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@17/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@17/Reference.java:253)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@17/Reference.java:215)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=4.12ms elapsed=64404.24s tid=0x00007fb1d45e8770 nid=0x5bef in Object.wait()  [0x00007fb1d91fb000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@17/Native Method)
	- waiting on <0x00000000f81375c0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@17/ReferenceQueue.java:155)
	- locked <0x00000000f81375c0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@17/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@17/Finalizer.java:172)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=1.00ms elapsed=64404.23s tid=0x00007fb1d45ef320 nid=0x5bf0 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=22.36ms elapsed=64404.23s tid=0x00007fb1d45f05b0 nid=0x5bf1 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Deflation Thread" #6 daemon prio=9 os_prio=0 cpu=1244.97ms elapsed=64404.23s tid=0x00007fb1d45f19a0 nid=0x5bf2 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=25020.58ms elapsed=64404.23s tid=0x00007fb1d45f3340 nid=0x5bf3 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #8 daemon prio=9 os_prio=0 cpu=4413.57ms elapsed=64404.23s tid=0x00007fb1d45f48b0 nid=0x5bf4 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #9 daemon prio=9 os_prio=0 cpu=117.37ms elapsed=64404.23s tid=0x00007fb1d45f5ce0 nid=0x5bf5 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Notification Thread" #10 daemon prio=9 os_prio=0 cpu=0.16ms elapsed=64404.22s tid=0x00007fb1d46152d0 nid=0x5bf6 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #11 daemon prio=8 os_prio=0 cpu=56.24ms elapsed=64404.22s tid=0x00007fb1d4617f50 nid=0x5bf8 in Object.wait()  [0x00007fb1d877e000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@17/Native Method)
	- waiting on <no object reference available>
	at java.lang.ref.ReferenceQueue.remove(java.base@17/ReferenceQueue.java:155)
	- locked <0x00000000f8137820> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@17/CleanerImpl.java:140)
	at java.lang.Thread.run(java.base@17/Thread.java:833)
	at jdk.internal.misc.InnocuousThread.run(java.base@17/InnocuousThread.java:162)

"surefire-forkedjvm-command-thread" #13 daemon prio=5 os_prio=0 cpu=447.93ms elapsed=64403.92s tid=0x00007fb1d4984dc0 nid=0x5bf9 runnable  [0x00007fb1d867d000]
   java.lang.Thread.State: RUNNABLE
	at java.io.FileInputStream.readBytes(java.base@17/Native Method)
	at java.io.FileInputStream.read(java.base@17/FileInputStream.java:276)
	at java.io.BufferedInputStream.read1(java.base@17/BufferedInputStream.java:282)
	at java.io.BufferedInputStream.read(java.base@17/BufferedInputStream.java:343)
	- locked <0x00000000f8254b10> (a java.io.BufferedInputStream)
	at java.io.BufferedInputStream.fill(java.base@17/BufferedInputStream.java:244)
	at java.io.BufferedInputStream.read1(java.base@17/BufferedInputStream.java:284)
	at java.io.BufferedInputStream.read(java.base@17/BufferedInputStream.java:343)
	- locked <0x00000000f8254b38> (a java.io.BufferedInputStream)
	at org.apache.maven.surefire.api.util.internal.Channels$3.readImpl(Channels.java:214)
	at org.apache.maven.surefire.api.util.internal.AbstractNoninterruptibleReadableChannel.read(AbstractNoninterruptibleReadableChannel.java:54)
	at org.apache.maven.surefire.booter.spi.LegacyMasterProcessChannelDecoder.decode(LegacyMasterProcessChannelDecoder.java:80)
	at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:343)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"helidon-client-0" #22 daemon prio=5 os_prio=0 cpu=4155.14ms elapsed=64399.87s tid=0x00007fb1d50ca150 nid=0x5c04 runnable  [0x00007fb1d847a000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@17/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17/SelectorImpl.java:129)
	- locked <0x00000000f8301d60> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000f8301d10> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@17/SelectorImpl.java:146)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.helidon.common.context.Contexts.runInContext(Contexts.java:117)
	at io.helidon.common.context.ContextAwareExecutorImpl.lambda$wrap$7(ContextAwareExecutorImpl.java:154)
	at io.helidon.common.context.ContextAwareExecutorImpl$$Lambda$760/0x0000000800eebcd8.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17/ThreadPoolExecutor.java:1136)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"pool-4-thread-1" #97 prio=5 os_prio=0 cpu=5.42ms elapsed=64394.53s tid=0x00007fb15c0ace90 nid=0x5c53 waiting on condition  [0x00007fb1bd770000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17/Native Method)
	- parking to wait for  <0x00000000f9057590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17/ForkJoinPool.java:3463)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17/ForkJoinPool.java:3434)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17/AbstractQueuedSynchronizer.java:1623)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17/ScheduledThreadPoolExecutor.java:1170)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"nioEventLoopGroup-48-1" #131 prio=10 os_prio=0 cpu=2.30ms elapsed=64348.29s tid=0x00007fb1d4f06520 nid=0x5c93 runnable  [0x00007fb1bd66f000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@17/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17/SelectorImpl.java:129)
	- locked <0x00000000ffd17b50> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ffd178d0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@17/SelectorImpl.java:146)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"nioEventLoopGroup-49-1" #132 prio=10 os_prio=0 cpu=88.56ms elapsed=64348.28s tid=0x00007fb16400e530 nid=0x5c94 runnable  [0x00007fb1d8379000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@17/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@17/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17/SelectorImpl.java:129)
	- locked <0x00000000ffd1b8b0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ffd1b630> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@17/SelectorImpl.java:146)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"pool-6-thread-1" #133 prio=5 os_prio=0 cpu=3.27ms elapsed=64348.18s tid=0x00007fb1600a9c60 nid=0x5c95 waiting on condition  [0x00007fb1bd56e000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17/Native Method)
	- parking to wait for  <0x00000000ffd0b1c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17/ForkJoinPool.java:3463)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17/ForkJoinPool.java:3434)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17/AbstractQueuedSynchronizer.java:1623)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17/ScheduledThreadPoolExecutor.java:1170)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17/Thread.java:833)

"Attach Listener" #134 daemon prio=9 os_prio=0 cpu=0.34ms elapsed=0.10s tid=0x00007fb190000a90 nid=0xd1a waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=1466.53ms elapsed=64404.25s tid=0x00007fb1d45e3560 nid=0x5bed runnable  

"GC Thread#0" os_prio=0 cpu=552.68ms elapsed=64404.26s tid=0x00007fb1d4244800 nid=0x5be8 runnable  

"GC Thread#1" os_prio=0 cpu=559.92ms elapsed=64403.91s tid=0x00007fb1a4004410 nid=0x5bfa runnable  

"G1 Main Marker" os_prio=0 cpu=14.52ms elapsed=64404.26s tid=0x00007fb1d4246dd0 nid=0x5be9 runnable  

"G1 Conc#0" os_prio=0 cpu=1584.68ms elapsed=64404.26s tid=0x00007fb1d4247d30 nid=0x5bea runnable  

"G1 Refine#0" os_prio=0 cpu=0.90ms elapsed=64404.26s tid=0x00007fb1d44a6570 nid=0x5beb runnable  

"G1 Service" os_prio=0 cpu=8124.04ms elapsed=64404.26s tid=0x00007fb1d44a7460 nid=0x5bec runnable  

"VM Periodic Task Thread" os_prio=0 cpu=23134.48ms elapsed=64404.22s tid=0x00007fb1d4616c20 nid=0x5bf7 waiting on condition  

JNI global refs: 17, weak refs: 0

@spericas
Copy link
Member

spericas commented Mar 2, 2022

Let's re-open if we see the failure again.

@spericas spericas closed this as completed Mar 2, 2022
@trentjeff
Copy link
Member

Seeing this test fail while building #3978

@trentjeff trentjeff reopened this Mar 21, 2022
@barchetta
Copy link
Member

barchetta commented Apr 13, 2022 •
edited
Loading

I'm seeing this more often in helidon-2.x when upgrading netty to 4.1.76.Final in my local repo. The test fails with java.net.SocketTimeoutException:

java.net.SocketTimeoutException: Read timed out
	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
	at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.base/java.io.InputStreamReader.read(InputStreamReader.java:181)
	at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
	at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
	at io.helidon.webserver.utils.SocketHttpClient.receive(SocketHttpClient.java:266)
	at io.helidon.webserver.HttpPipelineTest.testPipelining(HttpPipelineTest.java:116)

I did not see this in helidon 3 (master). Here is the test output (I added the request number to one of the log output -- that's why it might look slightly different than what is in the helidon-2.x branch):

2022.04.13 11:16:26 INFO io.helidon.webserver.NettyWebServer Thread[nioEventLoopGroup-62-1,10,main]: Channel '@default' started: [id: 0xed5f5988, L:/127.0.0.1:61259]
2022.04.13 11:16:26 INFO io.helidon.webserver.HttpPipelineTest Thread[main,5,main]: Started server at: https://localhost:61259
2022.04.13 11:16:26 INFO io.helidon.webserver.HttpPipelineTest Thread[main,5,main]: put client Thread[main,5,main]
2022.04.13 11:16:27 INFO io.helidon.webserver.HttpPipelineTest Thread[nioEventLoopGroup-63-1,10,main]: put server Thread[nioEventLoopGroup-63-1,10,main]
2022.04.13 11:16:27 INFO io.helidon.webserver.HttpPipelineTest Thread[nioEventLoopGroup-63-1,10,main]: put: / Thread[nioEventLoopGroup-63-1,10,main]
2022.04.13 11:16:27 INFO io.helidon.webserver.HttpPipelineTest Thread[main,5,main]: request0 client Thread[main,5,main]
2022.04.13 11:16:27 INFO io.helidon.webserver.HttpPipelineTest Thread[nioEventLoopGroup-63-1,10,main]: get server 0 Thread[nioEventLoopGroup-63-1,10,main]
2022.04.13 11:16:27 INFO io.helidon.webserver.HttpPipelineTest Thread[nioEventLoopGroup-63-1,10,main]: get server 1 Thread[nioEventLoopGroup-63-1,10,main]
2022.04.13 11:16:27 INFO io.helidon.webserver.HttpPipelineTest Thread[pool-305-thread-1,5,main]: get server schedule 1 Thread[pool-305-thread-1,5,main]
2022.04.13 11:16:28 INFO io.helidon.webserver.HttpPipelineTest Thread[pool-305-thread-1,5,main]: get server schedule 0 Thread[pool-305-thread-1,5,main]
2022.04.13 11:16:28 INFO io.helidon.webserver.HttpPipelineTest Thread[main,5,main]: request1 client Thread[main,5,main]
2022.04.13 11:16:38 INFO io.helidon.webserver.NettyWebServer Thread[nioEventLoopGroup-62-1,10,main]: Channel '@default' closed: [id: 0xed5f5988, L:/127.0.0.1:61259]

I think this output looks OK. We do a PUT and two GETs. The server sends the GET responses out of order (that's what we are testing). The client reads the first GET response and it's correct (0), but it blocks when it reads the second response.

This was referenced Apr 19, 2022
Merged
Merged
@barchetta
Copy link
Member

The above PRs seem to fix the issue (fingers crossed). Closing.

@barchetta barchetta self-assigned this Apr 19, 2022
@m0mus m0mus added this to Backlog Aug 12, 2024
@m0mus m0mus moved this to Closed in Backlog Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spericas spericas

@barchetta barchetta

Labels
intermittent P3 testing
Projects
Backlog
Archived in project
Milestone
No milestone
Development

No branches or pull requests
6 participants
@ljnelson @romain-grecourt @spericas @barchetta @m0mus @trentjeff