Multiple Test Failures from Blocked accept0 Syscalls (Debian CI Runs only)

[original-brownbear]

The following build: https://scans.gradle.com/s/4prwec7zf6pba/ failed in a very strange way.
Seemingly all nodes of the internal test cluster keep getting stuck in in accept calls on non-blocking server sockets.
The build log is full of failed connections and the following stuck thread reporting:

1> [2019-06-19T16:22:22,985][WARN ][o.e.t.n.MockNioTransport ] [node_sm1] Potentially blocked execution on network thread [elasticsearch[node_sm1][transport_worker][T#2]] [2401 milliseconds]:
--
1> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
1> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
1> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
1> org.elasticsearch.nio.ChannelFactory$RawChannelFactory$$Lambda$1624/1917852998.run(Unknown Source)
1> java.security.AccessController.doPrivileged(Native Method)
1> org.elasticsearch.nio.ChannelFactory$RawChannelFactory.accept(ChannelFactory.java:223)
1> org.elasticsearch.nio.ChannelFactory$RawChannelFactory.acceptNioChannel(ChannelFactory.java:180)
1> org.elasticsearch.nio.ChannelFactory.acceptNioChannel(ChannelFactory.java:55)
1> org.elasticsearch.nio.ServerChannelContext.acceptChannels(ServerChannelContext.java:47)
1> org.elasticsearch.nio.EventHandler.acceptChannel(EventHandler.java:45)
1> org.elasticsearch.transport.nio.TestEventHandler.acceptChannel(TestEventHandler.java:51)
1> org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:227)
1> org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:172)
1> org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129)
1> org.elasticsearch.nio.NioSelectorGroup$$Lambda$1545/393896253.run(Unknown Source)
1> java.lang.Thread.run(Thread.java:748)

It is not immediately clear to me how we could get into these calls blocking. It doesn't seem to be dead locks on some selector lock since no thread leaks are reported on the failing tests (though it could be that us interruption the node's thread pools clears all the stuck sys calls up). So far this seems to be a one time thing as far as I can see.

[elasticmachine]

Pinging @elastic/es-distributed

[henningandersen]

I am not sure, but this might be related to: #37543, where we also had threads hanging inside accept for no apparent reason. #39031 tried to improve this, but there were still issues after that.

[original-brownbear]

@henningandersen it seems the linked issue only affects 6.x though right? This happened on 7.x where we have a different network implementation based on non-blocking IO.

That said, I'm wondering if our close logic that potentially closes channels outside the transport thread cloud be at fault here again ... it seems it is possible for us to close the selector on a non-transport thread, which may lock when simultaneously accepting a connection ... what is very weird about this though is it happening across multiple threads :D If this was just one thread that might be a thing ... but all transport threads locking up simultaneously is hard to explain that way?

[original-brownbear]

One interesting find regarding the close behavior I came across now is that if I block for like 10s in org.elasticsearch.common.concurrent.CompletableContext#addListener (inside the listener) then my selector loop goes into a stat where it can't even be interrupted it seems:

Jun 19, 2019 9:29:22 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
WARNING: Will linger awaiting termination of 1 leaked thread(s).
Jun 19, 2019 9:29:27 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
SEVERE: 1 thread leaked from SUITE scope at org.elasticsearch.action.admin.HotThreadsIT: 
   1) Thread[id=36, name=elasticsearch[node_sc3][transport_worker][T#2], state=RUNNABLE, group=TGRP-HotThreadsIT]
        at java.base@12/sun.nio.ch.KQueue.poll(Native Method)
        at java.base@12/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)
        at java.base@12/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:162)
        at app//org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129)
        at app//org.elasticsearch.nio.NioSelectorGroup$$Lambda$1602/0x00000008016cd840.run(Unknown Source)
        at java.base@12/java.lang.Thread.run(Thread.java:835)
Jun 19, 2019 9:29:27 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
INFO: Starting to interrupt leaked threads:
   1) Thread[id=36, name=elasticsearch[node_sc3][transport_worker][T#2], state=RUNNABLE, group=TGRP-HotThreadsIT]
Jun 19, 2019 9:29:30 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
SEVERE: There are still zombie threads that couldn't be terminated:
   1) Thread[id=36, name=elasticsearch[node_sc3][transport_worker][T#2], state=RUNNABLE, group=TGRP-HotThreadsIT]
        at java.base@12/sun.nio.ch.IOUtil.drain(Native Method)
        at java.base@12/sun.nio.ch.KQueueSelectorImpl.clearInterrupt(KQueueSelectorImpl.java:298)
        at java.base@12/sun.nio.ch.KQueueSelectorImpl.processEvents(KQueueSelectorImpl.java:231)
        at java.base@12/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:139)
        at java.base@12/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:162)
        at app//org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129)
        at app//org.elasticsearch.nio.NioSelectorGroup$$Lambda$1602/0x00000008016cd840.run(Unknown Source)
        at java.base@12/java.lang.Thread.run(Thread.java:835)

com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.elasticsearch.action.admin.HotThreadsIT: 
   1) Thread[id=36, name=elasticsearch[node_sc3][transport_worker][T#2], state=RUNNABLE, group=TGRP-HotThreadsIT]
        at java.base@12/sun.nio.ch.KQueue.poll(Native Method)
        at java.base@12/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)
        at java.base@12/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:162)
        at app//org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129)
        at app//org.elasticsearch.nio.NioSelectorGroup$$Lambda$1602/0x00000008016cd840.run(Unknown Source)
        at java.base@12/java.lang.Thread.run(Thread.java:835)

	at __randomizedtesting.SeedInfo.seed([A17BF834B35181F4]:0)

REPRODUCE WITH: ./gradlew null --tests "org.elasticsearch.action.admin.HotThreadsIT" -Dtests.seed=A17BF834B35181F4 -Dtests.locale=en-DE -Dtests.timezone=Europe/Berlin

com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=36, name=elasticsearch[node_sc3][transport_worker][T#2], state=RUNNABLE, group=TGRP-HotThreadsIT]
        at java.base@12/sun.nio.ch.IOUtil.drain(Native Method)
        at java.base@12/sun.nio.ch.KQueueSelectorImpl.clearInterrupt(KQueueSelectorImpl.java:298)
        at java.base@12/sun.nio.ch.KQueueSelectorImpl.processEvents(KQueueSelectorImpl.java:231)
        at java.base@12/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:139)
        at java.base@12/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@12/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:162)
        at app//org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:129)
        at app//org.elasticsearch.nio.NioSelectorGroup$$Lambda$1602/0x00000008016cd840.run(Unknown Source)
        at java.base@12/java.lang.Thread.run(Thread.java:835)

	at __randomizedtesting.SeedInfo.seed([A17BF834B35181F4]:0)

(note that this is OSX though and KQueue might show different behavior here than EPoll). I'll have to look into this some more, though a short pause causing this isn't so great either since we can always have those on CI ... still this might not be related

[henningandersen]

Yeah, I am also in doubt, but find multiple overlapping symptoms:

• Happened to multiple threads in the JVM.
• Accept involved
• Close maybe involved here too.

But clearly there are many differences (network implementation, blocking vs non-blocking and more). I am not saying it is related, but wanted to bring this up in case it brings about new ideas of where the problem lies.

[original-brownbear]

Happened to multiple threads in the JVM.

This is by far the weirdest and most suspicious part. Those threads each come with their own selector and I don't see those transport threads sharing any monitor either ever ... but I also don't know of any system/kernel/... issue that would cause this behavior.

[DaveCTurner]

Other instances:

https://scans.gradle.com/s/xvdtt4bhikeye
https://scans.gradle.com/s/uqz54z4tqv6pa

[original-brownbear]

#43424 should help track this down quickly I hope :)

[henningandersen]

The two new and the original failure are all on debian 8, just like the similar failures in #24457.

[original-brownbear]

@atorok @mark-vieira do you know if anything changed about our Debian 8 environment lately that would explain this issue now coming up? (change in Kernel version or generally the environment coming back).

EDIT: I guess this is also a function of the CPUs used ... probably impossible to track down in a Cloud env ... sorry for the noise