Dealing with memory leak when unloading Sentry in web application

[tiguchi]

I'm not sure whether this should be a bug ticket instead, or whether I'm just blind and missing something. That's why I post this as a discussion / question.

TL;DR

• Sentry leaks Hub objects stored in the currenHub ThreadLocal that are bound to web application request threads (and background worker threads)
• (not sure) It looks like Sentry SDK does not provide a public method for removing these ThreadLocals from request / worker threads
  • Sentry.close() method cannot be used, since it completely shuts down the Sentry integration for entire application
• Tomcat complains about that during redeploy

Long Version

I have a problem with the latest Sentry SDK used in a JAX-RS web application (Apache CXF + Spring on Tomcat 9). When the application is reloaded due to a redeploy, I get repeatedly the following warning:

The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7fe03bb]) and a value of type [io.sentry.Hub] (value [io.sentry.Hub@5f452f8e]) but failed to remove it when the web application was stopped.

I found the following PR that was supposed to fix this problem:

#1909

After a bit of debugging and poking around, I have a feeling this is caused by Sentry being used within request threads (either for capturing error events, or for performance analytics). The Sentry singleton has a thread local variable for keeping separate Hub instances per thread. These thread local hub copies are never cleaned up.

I implemented an application context listener that calls Sentry.close() when the context is closed. The Sentry.close method does remove the hub threadlocal. But only the one for the thread that is responsible for closing the servlet context. The Sentry Hub instances that were in use by the request threads are not removed.

I could set up a request listener that would try to clean up the Sentry hub when a request has finished, but that would probably defy the purpose of caching those hubs per thread, and eventually come with some overhead of recreating them?

Also, if I would call Sentry.close() after a request has finished, then this would shut down the entire Sentry integration, and the global hub, so given the current Sentry SDK implementation, that doesn't seem to be possible. There needs to be a public API for only removing the currentHub thread local for the current thread, without unregistering everything.

Even if the Sentry SDK had a mechanism for just removing the hub for the current thread, I don't know if Java's servlet API has a mechanism for running cleanup code when request threads are shut down.

I'm a bit at a loss here, and I would like to know how to correctly integrate and configure Sentry so it doesn't leak memory over time

[adinauer]

@maciejwalkowiak will investigate if/how Tomcat shares recources between web apps.
@tiguchi we'll get back to you once we know more.