[native-image] Finding thread started through static initializers?

[sundbp]

Hi,

I'm trying to understand how I work out which static initializer leads to a thread being started as per the following error message from native-image:

error: Detected a started Thread in the image heap. This is not supported. The object was reached from a static initializer. All static class initialization is done during native image construction, thus a static initializer cannot contain code that captures state dependent on the build machine. Write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Error: Detected a started Thread in the image heap. This is not supported. The object was reached from a static initializer. All static class initialization is done during native image construction, thus a static initializer cannot contain code that captures state dependent on the build machine. Write your own initialization methods and call them explicitly from your main entry point.
Trace: 	object java.lang.ref.WeakReference
	object io.netty.util.Recycler$Stack
	object java.lang.Object[]
	object io.netty.util.internal.InternalThreadLocalMap
	object io.netty.util.concurrent.FastThreadLocal$1
	object io.netty.util.internal.ObjectCleaner$AutomaticCleanerReference
	object java.util.concurrent.ConcurrentHashMap$Node
	object java.util.concurrent.ConcurrentHashMap$Node[]
	object java.util.concurrent.ConcurrentHashMap
	object io.netty.util.internal.ConcurrentSet
	method io.netty.util.internal.ObjectCleaner.register(Object, Runnable)
Call path from entry point to io.netty.util.internal.ObjectCleaner.register(Object, Runnable):
	at io.netty.util.internal.ObjectCleaner.register(ObjectCleaner.java:95)
	at io.netty.util.concurrent.FastThreadLocal.registerCleaner(FastThreadLocal.java:163)
	at io.netty.util.concurrent.FastThreadLocal.get(FastThreadLocal.java:147)
	at io.netty.channel.ChannelOutboundBuffer.clearNioBuffers(ChannelOutboundBuffer.java:357)
	at io.netty.channel.ChannelOutboundBuffer.close(ChannelOutboundBuffer.java:687)
	at io.netty.channel.ChannelOutboundBuffer$3.run(ChannelOutboundBuffer.java:654)
	at java.lang.Thread.run(Thread.java:748)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:222)
	at Lcom/oracle/svm/core/code/CEntryPointCallStubs;.com_002eoracle_002esvm_002ecore_002eposix_002ethread_002ePosixJavaThreads_002epthreadStartRoutine_0028com_002eoracle_002esvm_002ecore_002eposix_002ethread_002ePosixJavaThreads_0024ThreadStartData_0029(generated:0)

I'm digging around the classes mentioned in that trace but it's not at all clear to me what static initialiser takes us down this path. How would an experienced SVM user go about this case?

Thanks!

[cstancu]

The two parts of the error message should be interpreted like this. First the trace:

Trace: 	object java.lang.ref.WeakReference
	object io.netty.util.Recycler$Stack
	object java.lang.Object[]
	object io.netty.util.internal.InternalThreadLocalMap
	object io.netty.util.concurrent.FastThreadLocal$1
	object io.netty.util.internal.ObjectCleaner$AutomaticCleanerReference
	object java.util.concurrent.ConcurrentHashMap$Node
	object java.util.concurrent.ConcurrentHashMap$Node[]
	object java.util.concurrent.ConcurrentHashMap
	object io.netty.util.internal.ConcurrentSet
	method io.netty.util.internal.ObjectCleaner.register(Object, Runnable)

is generated during scanning of heap roots, i.e., scanning of constants. It represents a chain of concrete objects. The root of this chain is a constant embeded in the method io.netty.util.internal.ObjectCleaner.register(Object, Runnable) and it has the type object io.netty.util.internal.ConcurrentSet , and the leaf, i.e., the one that references the Thread object, is object java.lang.ref.WeakReference.

The second part

Call path from entry point to io.netty.util.internal.ObjectCleaner.register(Object, Runnable):
	at io.netty.util.internal.ObjectCleaner.register(ObjectCleaner.java:95)
	at io.netty.util.concurrent.FastThreadLocal.registerCleaner(FastThreadLocal.java:163)
	at io.netty.util.concurrent.FastThreadLocal.get(FastThreadLocal.java:147)
...

is the point in the static analysis where the error was encountered. The important bit of information there is the tip of the call path: io.netty.util.internal.ObjectCleaner.register(Object, Runnable). That's where the constant that is the root of the object chain is detected.

Looking at the code for io.netty.util.internal.ObjectCleaner.register(Object, Runnable) the private static final Set<AutomaticCleanerReference> LIVE_SET = new ConcurrentSet<AutomaticCleanerReference>(); field is the only one that matches the error parameters. In ObjectCleaner.register() an AutomaticCleanerReference is added to this set and the AutomaticCleanerReference extends WeakReference and contains a Runnable, which is the Runnable cleanupTask parameter to ObjectCleaner.register().

Looking in the IDE ObjectCleaner.register() is called from multiple static contexts. Any of those can be a candidate for where it is reached from. The Call path from entry point bit of information above gives you the shortest such path between the application entry point and ObjectCleaner.register(). However it doesn't give you the static initializer where it is called from and from where the Runnable instance is passed in.

We need to work on improving the trace in such cases. Thanks for reporting.

[sundbp]

Ok, I understand the output a lot better now.

It starts to sound like this may be tricky to work around without making changes to netty itself with regards to these static initializers - or am I getting that wrong and you can think of ways to continue?

(fwiw, this happens if I add [io.netty/netty-tcnative-boringssl-static "2.0.7.Final"] to the same cut-down example repo as my other thread, not code changes, just by adding it to the classpath. Eventually I'd need to call gRPC behind SSL in my cli tool so was giving that a spin and ran into this.)

[cstancu]

I haven't looked much into the static initializers in netty yet. At this point I cannot think of any quick ways to continue, it will probably require a bit more work on our side first. We are also considering the possibility to push some of the static initializers at runtime to deal with situations like this, trading some of the benefits of pre-initializing the entire heap with easing the development and incrementally refactoring libraries code. As for SSL we are still working on that, but it should be ready soon.

[sundbp]

Ok - thanks for the info. Let me know if there's anything I can do to help test things as you continue forward!

[cstancu]

The approach to static initialization has greatly evolved since this issue was reported and the error messages have improved. Please refer to this document for further details https://github.com/oracle/graal/blob/master/substratevm/CLASS-INITIALIZATION.md.