Shutdown is hanging #38164
Comments
quarkus-bot
bot
added
the
area/securepipeline
|
Unfortunately a simple change to address a possible race condition in narayana did not address this. I was hoping some more eyes could help determine if the remaining element can be forcibly terminated in this case. |
|
@shawkins Can you add some thread dumps so that we can see which threads are waiting? |
|
@mmusgrov other than the main thread being held at shutdown as shown in https://issues.redhat.com/browse/JBTM-3830 there are no other threads doing work - see the attached for the full thread dump.
|
Karm
removed
the
area/securepipeline
|
To follow up the underlying issue is with TransactionReaper.terminate(false) - instead of immediately shutting down, the original timeout of any lingering transactions is still being honored. Thus the shutdown will hang until the transaction(s) times out. The lingering transaction is not from the liquibase migration as mentioned above, but is from a transaction that is managed by Keycloak's TransactionalSessionHandler - https://github.com/keycloak/keycloak/blob/ed96b13312198569b7e68647eefe44d5ee24c739/quarkus/runtime/src/main/java/org/keycloak/quarkus/runtime/transaction/TransactionalSessionHandler.java#L27 In the affected test cases the server does not complete a normal startup, and a session that was started does not have the corresponding close method called prior to TransactionReaper.terminate. So a question on the quarkus/jaxrs side - should there be a guarentee that our CloseSessionHandler is called? https://github.com/keycloak/keycloak/blob/ed96b13312198569b7e68647eefe44d5ee24c739/quarkus/runtime/src/main/java/org/keycloak/quarkus/runtime/integration/jaxrs/CloseSessionHandler.java#L68 cc @geoand |
|
Not sure I undertstand, where were you hoping |
|
@geoand both our create and close are expected to be called via the handler chain on QuarkusResteasyReactiveRequestContext. |
|
I admit I have not understood the details of the issue, but I don't see why you expect that would be called when no HTTP requests have been handled and therefore there is no REST layer code involved. |
see the correction in #38164 (comment) - there is REST session being created which is responsible for the left-over transaction. |
Is this a Keycloak thing? |
Keycloak is defining a CreateSessionHandler [1] and a CloseSessionHandler [2] which get installed as handlers / filters in the QuarkusResteasyReactiveRequestContext [3]. When transactions are cleaned up appropriately I can see paired calls to the create and close methods via their respective handlers - for example: The KeycloakSession, which is put into the routing context by the CreateSessionHandler, is implemented by a DefaultKeycloakSession [4] and provides access to the underlying transaction manager. [1] https://github.com/keycloak/keycloak/blob/5373f3c97a801c000fb0ac6b54b2148b1e6c276b/quarkus/runtime/src/main/java/org/keycloak/quarkus/runtime/integration/resteasy/CreateSessionHandler.java |
|
Just to clarify things a little: Handlers and Filters are different things. The former are a RESTEasy Reactive concept that essentially comprise the basic unit of an HTTP request handling pipeline - extensions can contribute to these as Keycloak does. With that it mind, it seems like If I understand correctly, you start a transaction when an HTTP request is being processed and you want to close it when the processing is complete, correct? |
Yes that is the expectation.
That's not clear to me either yet. There are some other stacks that end up in the logs, such as #26604, but even with resolving that another stack pops up - and the orphaned transaction problem persists. I was hoping there was something simple / guarenteed here about how close runs. |
|
After clearing up the #26604 here's what the log looks like: Then a log of transaction begin via TransactionSessionHandler: Then a bunch of repeating errors, but no call to CloseSessionHandler.close: The end result is that this transaction will cause the transaction reaper to be stuck at shutdown holding the main thread (until https://issues.redhat.com/browse/JBTM-3830 is addressed): |
|
Is there any simple way for me to reproduce this? |
|
Other than repeatively running our test case, no I don't have a simple reproducer. For reference the exception that is expected to cause premature shutdown / abnormal startup is: |
|
But in that case I still don't see how an HTTP request is being handled as the HTTP server has not come up |
|
@geoand right that is confusing. Trapping this a bit more in the debugger, I can see resteasy reactive logic adding the QuarkusResteasyReactiveRequestContext as a task - at the same time the malformed sequence is being thrown from starting the HTTP server. Then when that task runs, it triggers the creation of the transaction. Stack for adding the task: |
|
It seems to me like some HTTP request is trying to be served before everything is up and running |
Yes, I'll try the scenario again. I think the cause of the orphaned transaction was identified as well - keycloak/keycloak#27682 |
|
No reoccurances in our test case after 50 runs. Instead I see logs like: Which would not have occurred when exhibiting the hanging behavior. |
|
👌 |
Describe the bug
See https://issues.redhat.com/browse/JBTM-3830 and keycloak/keycloak#23786 - for some of tests where the server has a failed start up, the process will fail to exit due to the TransactionReaper shutdown not completing.
Expected behavior
The keycloak server should stop.
Actual behavior
The keycloak server shutdown hangs.
How to Reproduce?
Add a RepeatedTest annotation to https://github.com/keycloak/keycloak/blob/5d3df39f04d64bcfe3acc077de837a57bd838b64/quarkus/tests/integration/src/test/java/org/keycloak/it/cli/dist/FipsDistTest.java#L124 - this only occurs about 10% of the time. In the output when it is hanging the test will capture the thread dump.
Output of
uname -aorverNo response
Output of
java -versionNo response
Quarkus version or git rev
3.2.9
Build tool (ie. output of
mvnw --versionorgradlew --version)No response
Additional information
No response
The text was updated successfully, but these errors were encountered: