Process instance no longer subscribed to message after unsuccessful process instance migration

[korthout]

Describe the bug

We've discovered a critical bug that can delete the ProcessMessageSubscription and MessageSubscription during Process Instance Migration, even though the migration is rejected. This should not have any affects and all changes are supposed to have been rolled back.

https://jira.camunda.com/browse/SUPPORT-24591

To Reproduce

• Create an instance of a process that subscribes to a message catch event
• Migrate the process instance to a process with the same message catch event, without mapping the message catch event
• Notice that the process instance migration is rejected
  • "Expected to migrate process instance '2251799813685253' but active element with id 'Activity_04uuugc' attempts to subscribe to a message it is already subscribed to with name 'PutOnHold'. Migrating active elements that subscribe to a message they are already subscribed to is not possible yet. Please provide a mapping instruction to message catch event with id 'Event_0qc5a9d' to migrate the respective message subscription."
• After at least 30 seconds, publish a message
• Notice that the message is not correlated

To recover: retry the process instance migration after waiting at least 30 seconds. This will automatically recreate the deleted message subscription as documented:

Add catch events: if a catch event of the target process is not the target of a mapping instruction, then a new subscription is opened during migration.

Expected behavior

When the migration is rejected, the process instance should not have been changed.

Log/Stacktrace

Full Stacktrace

..
13:35:16 C PI_MIGRATION MIGRATE        - #27-> -1 K04 - ProcessInstanceMigrationRecord {"targetProcessDefinitionKey":2251799813685251,"mappingInstructions":[{"sourceElementId":"A","targetElementId":"A"}],"processInstanceKey":2251799813685252}
13:35:16 R PI_MIGRATION MIGRATE        - #28->#27 K04 - ProcessInstanceMigrationRecord {"targetProcessDefinitionKey":2251799813685251,"mappingInstructions":[{"sourceElementId":"A","targetElementId":"A"}],"processInstanceKey":2251799813685252} !INVALID_STATE (Expected to migrate process instance '2251799813685252' but active element with id 'A' attempts to subscribe to a message it is already subscribed to with name 'message'. Migrating active elements that subscribe to a message they are already subscribed to is not possible yet. Please provide a mapping instruction to message catch event with id 'boundary' to migrate the respective message subscription.)
13:35:46 C MSG_SUB      DELETE         - #29-> -1  -1 - "message" (inter.) @[K09] in <process ?[K04]> (no vars)
13:35:46 E MSG_SUB      DELETED        - #30->#29 K12 - "message" (inter.) correlationKey: key @[K09] in <process "process"[K04]> (no vars)
13:35:46 C PROC_MSG_SUB DELETE         - #31->#29  -1 - "message" (inter.) @[K09] in <process ?[K04]> (no vars) (tenant: <default>)
13:35:46 E PROC_MSG_SUB DELETED        - #32->#29 K10 - "message" (inter.) correlationKey: key @[K09] in <process "process"[K04]> (no vars) (tenant: <default>)

Environment:

• Zeebe Version: TBD, at least 8.6.x
• Slack channel: https://camunda.slack.com/archives/C081F9YLAF6

[korthout]

@berkaycanbc and I investigated this issue and have the following findings:

• the message subscription is DELETE command is written to the log some 30 seconds after the migration was rejected
• the PendingMessageSubscriptionChecker sends the DELETE command and runs every 30 seconds
• it sends the DELETE command because it finds the message subscription in the pending state
• it finds it in the transient state because it is not reverted on rejection

One solution could be to remove the pending subscription from this transient state when rejecting the instance migration (and reverting changes). However, that would mean that the checker would be able to encounter the pending subscription before it may still be reverted.

Instead, it would be better to avoid adding the pending subscription to the transient state until the entire command has been processed successfully.

We could try to achieve this using the stream processor's side effects, which are only applied after a command was successfully processed.

[korthout]

@berkaycanbc and I have been making progress on this issue (branch: korthout-berkay-25166-transient-message-state-not-transactional). We've found a solution to the problem by using a side-effect to make the transient state change.

We've verified that this works with our reproducing test case, however this test is slow and not maintainable, so we will open a PR as soon as we have one that is. We also need to resolve the issue on subscribing to new messages as this could also produce wrong results (although less impactful).

[lenaschoenburg]

@korthout Since it's possible to recover from this (at least that's how I understood your "To recover" instructions), I'd argue that this doesn't qualify as critical.

[korthout]

Good point @lenaschoenburg. I considered it critical because of the impact, but the workaround should be reasonable. Lowered to high.

[korthout]

Closing as it's fixed on all supported branches:

• [Backport main] Update transient message state as side-effect #25476
• [Backport release-8.7.0-alpha2] Update transient message state as side-effect #25489
• stable/8.6 Update transient message state as side-effect #25298
• [Backport stable/8.5] Update transient message state as side-effect #25497
• [Backport stable/8.4] Update transient message state as side-effect #25511
• [Backport stable/8.3] Update transient message state as side-effect #25532