Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wrong detection of heurestic transactions with WildFly 34 Beta #312

Closed
simkam opened this issue Oct 2, 2024 · 0 comments
Closed

wrong detection of heurestic transactions with WildFly 34 Beta #312

simkam opened this issue Oct 2, 2024 · 0 comments

Comments

@simkam
Copy link
Contributor

simkam commented Oct 2, 2024

In server I have HEURISTIC_ROLLBACK transaction

[standalone@localhost:9990 /] /subsystem=transactions/log-store=log-store:read-resource(recursive=true, include-runtime=true)
{
    "outcome" => "success",
    "result" => {
        "expose-all-logs" => false,
        "type" => "default",
        "transactions" => {"0:ffff0a83027b:49e508a7:66fbc17e:10" => {
            "age-in-seconds" => "6668",
            "id" => "0:ffff0a83027b:49e508a7:66fbc17e:10",
            "jmx-name" => undefined,
            "type" => "StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction",
            "participants" => {"java:/MockXAResource" => {
                "eis-product-name" => "MockXAResource Test",
                "eis-product-version" => "0.1.Mock",
                "jmx-name" => undefined,
                "jndi-name" => "java:/MockXAResource",
                "status" => "HEURISTIC_ROLLBACK",
                "type" => "/StateManager/AbstractRecord/XAResourceRecord"
            }}
        }}
    }
}

During scaledown, the operator should recognize this transaction and change the pod status to SCALING_DOWN_RECOVERY_HEURISTICS, but the pod is stuck in SCALING_DOWN_RECOVERY_PROCESSING. (The scale-down is still blocked, which is good.)

{"level":"info","ts":"2024-10-01T11:02:15.657Z","logger":"controllers.WildFlyServer","msg":"Reconciling WildFlyServer","name":"msimka-namespace/tx-client"}
{"level":"info","ts":"2024-10-01T11:02:15.657Z","logger":"wildflyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-client","Resource.Name":"tx-client"}
{"level":"info","ts":"2024-10-01T11:02:17.476Z","logger":"controllers.WildFlyServer","msg":"Transaction recovery scaledown processing","name":"msimka-namespace/tx-client","Pod Name":"tx-client-0","IP Address":"10.131.2.123","Pod State":"SCALING_DOWN_RECOVERY_PROCESSING","Pod's Recovery Counter":323}
{"level":"info","ts":"2024-10-01T11:02:19.235Z","logger":"controllers.WildFlyServer","msg":"Recovery properties at pod were already defined. Skipping server restart.","name":"msimka-namespace/tx-client","Pod Name":"tx-client-0"}
{"level":"info","ts":"2024-10-01T11:02:19.245Z","logger":"controllers.WildFlyServer","msg":"Executing the recovery scan for the pod","name":"msimka-namespace/tx-client","Pod name":"tx-client-0","Pod IP":"10.131.2.123","Recovery port":4712}
{"level":"info","ts":"2024-10-01T11:02:25.755Z","logger":"controllers.WildFlyServer","msg":"Executing the recovery scan for the pod","name":"msimka-namespace/tx-client","Pod name":"tx-client-0","Pod IP":"10.131.2.123","Recovery port":4712}
{"level":"info","ts":"2024-10-01T11:02:32.359Z","logger":"controllers.WildFlyServer","msg":"The pod is trying to recover unfinished transactions","name":"msimka-namespace/tx-client","Pod name":"tx-client-0","Message":"A recovery scan is needed as the log store of the pod tx-client-0 is not empty, transaction list: map[0:ffff0a83027b:49e508a7:66fbc17e:10:map[age-in-seconds:5387 id:0:ffff0a83027b:49e508a7:66fbc17e:10 jmx-name:<nil> participants:map[java:/MockXAResource:map[eis-product-name:MockXAResource Test eis-product-version:0.1.Mock jmx-name:<nil> jndi-name:java:/MockXAResource status:HEURISTIC_ROLLBACK type:/StateManager/AbstractRecord/XAResourceRecord]] type:StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction]]"}
{"level":"info","ts":"2024-10-01T11:02:32.359Z","logger":"controllers.WildFlyServer","msg":"The recovery counter of the pod is: 324","name":"msimka-namespace/tx-client","Pod name":"tx-client-0"}
{"level":"info","ts":"2024-10-01T11:02:32.359Z","logger":"wildflyserver_resources","msg":"Updating status of WildFlyServer","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-client"}
{"level":"info","ts":"2024-10-01T11:02:32.369Z","logger":"controllers.WildFlyServer","msg":"For statefulset scaling down need to verify if pods were cleaned by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-client"}
{"level":"info","ts":"2024-10-01T11:02:32.369Z","logger":"controllers.WildFlyServer","msg":"Statefulset was not scaled to the desired replica size 0 (current StatefulSet size: 1). Transaction recovery scaledown process has not cleaned all pods. Please, check status of the WildflyServer tx-client","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-client"}

It looks like the check /subsystem=transactions/log-store=log-store/transactions=*/participants=*:query(where={"status"="HEURISTIC"} is wrong. Threre is no HEURISTIC status. AFAIK there are HEURISTIC_COMMIT, HEURISTIC_ROLLBACK, HEURISTIC_HAZARD and HEURISTIC_MIXED.

JBoss CLI query doesn't support wildcards, so *:query(where={"status"="HEURISTIC"} returns nothing, but *:query(where={"status"="HEURISTIC_ROLLBACK"} returns the expected transaction

jmfinelli added a commit to jmfinelli/wildfly-operator that referenced this issue Oct 2, 2024
@simkam simkam changed the title wrong detection of heurestic transactions wrong detection of heurestic transactions with WildFly 34 Beta Oct 2, 2024
jmfinelli added a commit to jmfinelli/wildfly-operator that referenced this issue Oct 2, 2024
jmfinelli added a commit to jmfinelli/wildfly-operator that referenced this issue Oct 2, 2024
yersan added a commit that referenced this issue Oct 3, 2024
[#312] Wrong detection of heurestic transactions with WildFly 34 Beta
@yersan yersan closed this as completed Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants