Java: ErrorStreamSink does not clean itself up (invoked by mvn exec:java)

[bundgaard]

🐛 Bug Report

[WARNING] thread Thread[JsiiRuntime.ErrorStreamSink,5,<REDACTED>] was interrupted but is still alive after waiting at least 15000msecs
[WARNING] thread Thread[JsiiRuntime.ErrorStreamSink,5,<REDACTED>] will linger despite being asked to die via interruption
[WARNING] NOTE: 1 thread(s) did not finish despite being asked to  via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=<REDACTED>,maxpri=10]
java.lang.IllegalThreadStateException
    at java.lang.ThreadGroup.destroy (ThreadGroup.java:774)
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:293)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:564)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

Affected Languages

• TypeScript or Javascript
• Python
• Java
• .NET (C#, F#, ...)

General Information

• JSII Version: 1.20.0 (AWS-CDK 1.87.0)
• Platform: Linux/MacOS/Windows

What is the problem?

We started receiving this error yesterday, and the only way to get rid of this error is using -Dexec.cleanupDaemonThreads=false

Verbose Log

[RomainMuller]

Alright so basically I reckon this is due to the JsiiRuntime.ErrorStreamSink thread being blocked on reading from the child process' STDERR output... If that read is uninterruptible, then it might block "forever".

What's tricky here is that it is surprisingly hard in Java to make a "blocking read with timeout" happen without involving yet another thread, which might also get locked on uninterruptible blocking operations...

---

Additionally, I have not encountered this particular problem on macOS. The General Information in this issue mentions all OS', and so I would like to confirm you have actually encountered the same symptoms on all those OS'es... If not, can you please specify which OS'es you've confirmed the problem exists on?

[bundgaard]

Thank you for your answer, in our team we have a mix of Ubuntu users, Mac OS users and Windows Users (Windows + WSL), Vmware with Linux as guest, and we all experienced this problem, and I don't know what to provide with more information, as we settled on using the exec.cleanupDaemonThreads=false to skip the problem, this argument is from the Exec Maven plugin.

[RomainMuller]

@bundgaard - it turns out I was actually not attempting to reproduce this problem correctly. It may however not be possible to completely fix this problem (although I have a PR out that attempts to)...

The culprit here is that we have the ErrorStreamSink thread (in the current runtime) read out a child process' STDERR and process it out... That child process only gets explicitly terminated by a VM shutdown hook; and the ErrorStreamSink thread basically needs to live as long as that child process lives (if we do not consume it's output, it may block trying to write more data on a full buffer)... It would appear that the mvn exec:java workflow attempts to clean up the thread group before VM shutdown happens, and so it is a little awkward.

Now -- should my current PR not be sufficient to solve it, I have a couple of other ideas that we can try, in order to more gracefully integrate with exec:java, and I'll happily follow-up on the subject.

[bundgaard]

@RomainMuller Thanks for informing me, if I can provide you with anything, in respect to our company policy, I will gladly help you.

Keep up the good work.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.