pubsub async subscriber listener logging error on stop #2588
Assignees
Labels
api: pubsub
Issues related to the Pub/Sub API.
priority: p2
Moderately-important priority. Fix may not be included in next release.
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Comments
anthmgoogle
added
api: pubsub
|
@rcoy-v Would you be able to share a little more of the stack trace with us? It would be really helpful to see what's trying to execute. |
|
Stack trace: The stack trace is not coming from the subscriber, but a listener implementing We are running this with Spring Boot. The parts related to Google pubsub are still pretty basic, following close to the async example here. Testing this further, the error is logged if the app is stopped from SIGINT/SIGTERM. If I programmatically stop the subscriber after a sleep of 5 seconds, no error is logged. To be clear, |
|
I'm having the same problem with the latest version of pubsub 0.30.0-beta.. Here is my complete stacktract (don't mind the clojure code..) I can reproduce at any time by running a If you need any more info please let me know. |
|
@neuromantik33 This is rather surprising. In It is possible that you're providing an executor and shutting it down before calling |
|
@pongad As far as I know I'm not providing any executor save the one required by the (def ^:private direct-executor (MoreExecutors/directExecutor))
(defn- create-subscriber!
"Returns a new subscriber using the default configuration."
^Subscriber
[{:keys [project-id credentials channel-provider]} sub receiver]
(let [sub-name (make-sub-name project-id sub)
builder (cond-> (Subscriber/defaultBuilder sub-name receiver)
credentials (.setCredentialsProvider (c/fixed-credentials credentials))
channel-provider (.setChannelProvider channel-provider))]
(log/info "Creating subscriber:" (->clj sub-name))
(doto (.build builder)
(.addListener (proxy [ApiService$Listener] []
(failed [from ex]
(log/error ex "An unexpected error occurred")))
direct-executor))))
(defn shutdown!
"Shuts down the subscriber and waits for the latter to reach the terminated state."
[^Subscriber subscriber]
(log/info "Shutting down:" (->clj subscriber))
(-> subscriber .stopAsync .awaitTerminated)) |
|
I found the problem. I'll put the diagnosis in the PR description. |
pongad
added a commit
to googleapis/gax-java
that referenced
this issue
Dec 7, 2017
Fixes googleapis/google-cloud-java#2588 . Background ========== We previously decided that executors created from InstantiatingExecutorProvider should not block JVM from exiting if user forgets to call shutdown(). To implement this, we used Guava's MoreExecutors.getExitingScheduledExecutorService. It works by 1. making all executor threads daemon; the JVM exits after the last non-daemon thread exits, so these threads don't block termination 2. adding a shutdown hook; when shutting down, we create one *non-daemon* thread. This thread shuts down the executor, preventing more jobs from being added, then wait for the maximum of 2 minutes for existing jobs to complete. In either case, the thread simply exits. Since (hopefully) there are no non-daemon threads left, the JVM exits due to (1). Problem ======= Frameworks, like Spring, use shutdown hooks to gracefully exit. However, JVM runs shutdown hooks in unspecified order. The executor's shutdown might run first and shuts down the executor. Then, the graceful-exit logic might try to execute more tasks, causing RejectedExecutionExceptions and making graceful-exit not very graceful. This problem isn't isolated to shutdown hooks. For example, if we create two non-daemon threads A and B; A calls System.exit(); and B calls executor.execute(); B might also get the exception. Proposed Solution ================= This PR implements (1) but not (2). On exit, we'd no longer shuts down the executor. User code may still use executor to perform any shutdown logic it needs. However, when all user threads exit, executor threads will abruptly terminate with no grace period.
|
@pongad Is there a timeframe when this fix will be available? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have the same situation as described in #2485. Using version 0.26.0.
A listener for failed state is logging an error message when stopping an async subscriber:
Although this error is described as harmless in the referenced issue, it pollutes our logs during shutdown and deployments. The only seam I can find is detect if the
fromstate isSTOPPING, and ignore any failure. But I feel like this would potentially miss real errors if they were to occur during shutdown.Are there any suggested ways to better handle this?
The text was updated successfully, but these errors were encountered: