Unhandled NoSuchElementException when looking for executable process while deploying BPMN resource

[lenaschoenburg]

Describe the bug

 java.util.NoSuchElementException: No value present
	at java.util.Optional.orElseThrow(Unknown Source) ~[?:?]
	at io.camunda.zeebe.engine.state.deployment.DbProcessState.updateInMemoryState(DbProcessState.java:180) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]

To Reproduce
Unclear

Expected behavior

Invalid BPMN resources should be handled gracefully and not result in an unhandled NoSuchElementException.

Log/Stacktrace

Full Stacktrace

 java.util.NoSuchElementException: No value present
	at java.util.Optional.orElseThrow(Unknown Source) ~[?:?]
	at io.camunda.zeebe.engine.state.deployment.DbProcessState.updateInMemoryState(DbProcessState.java:180) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.engine.state.deployment.DbProcessState.lookupProcessByIdAndPersistedVersion(DbProcessState.java:314) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.engine.state.deployment.DbProcessState.getLatestProcessVersionByProcessId(DbProcessState.java:217) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.engine.processing.deployment.transform.BpmnResourceTransformer.transformProcessResource(BpmnResourceTransformer.java:138) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.engine.processing.deployment.transform.BpmnResourceTransformer.lambda$transformResource$0(BpmnResourceTransformer.java:77) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.util.Either$Right.map(Either.java:355) ~[zeebe-util-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.engine.processing.deployment.transform.BpmnResourceTransformer.lambda$transformResource$1(BpmnResourceTransformer.java:75) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.util.Either$Right.flatMap(Either.java:366) ~[zeebe-util-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.engine.processing.deployment.transform.BpmnResourceTransformer.transformResource(BpmnResourceTransformer.java:65) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.engine.processing.deployment.transform.DeploymentTransformer.transformResource(DeploymentTransformer.java:120) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.engine.processing.deployment.transform.DeploymentTransformer.transform(DeploymentTransformer.java:97) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.engine.processing.deployment.DeploymentCreateProcessor.processRecord(DeploymentCreateProcessor.java:96) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.engine.Engine.process(Engine.java:127) ~[zeebe-workflow-engine-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.stream.impl.ProcessingStateMachine.lambda$processCommand$3(ProcessingStateMachine.java:264) ~[zeebe-stream-platform-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransaction.run(ZeebeTransaction.java:84) ~[zeebe-db-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.stream.impl.ProcessingStateMachine.processCommand(ProcessingStateMachine.java:260) ~[zeebe-stream-platform-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.stream.impl.ProcessingStateMachine.tryToReadNextRecord(ProcessingStateMachine.java:209) ~[zeebe-stream-platform-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.stream.impl.ProcessingStateMachine.readNextRecord(ProcessingStateMachine.java:185) ~[zeebe-stream-platform-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:92) ~[zeebe-scheduler-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:45) ~[zeebe-scheduler-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:119) ~[zeebe-scheduler-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:106) ~[zeebe-scheduler-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:87) ~[zeebe-scheduler-8.2.0-alpha3.jar:8.2.0-alpha3]
	at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:198) ~[zeebe-scheduler-8.2.0-alpha3.jar:8.2.0-alpha3] 

Environment:

• SaaS
• Zeebe Version: 8.2.0-alpha3

Error group

[lenaschoenburg]

Might be related to #11392

[korthout]

We need to understand the impact before we can prioritize this. Let's investigate whether this leads to a blacklisted instance.

[remcowesterhoud]

I had a look into this with @koevskinikola. We suspect that somehow during the deployment, the deployment record process metadata does not contain a process that we do have in the bpmn file. As a result we have a process in the bpmn file that we have not stored as a PersistedProcess, resulting in this exception.

We are unsure how this could occur and how we can reproduce it. The exception has occurred 3 times in trial clusters thus far.

[remcowesterhoud]

We discussed within the team and we will have another look into this issue together during out next mob-programming hour 2 weeks from now.

[korthout]

Today, the team looked again at this issue. Having no way to reproduce it nor a way to find out how it might have happened makes this hard to debug.

I've had another look at several parts:

• ❌ could this have been caused by NPE during transformation of BPMN model #11392 by checking multiple scenarios (e.g. deploying valid processes with id's that were already used in deployments that encountered that bug)
• ❓ what is this code actually doing?

This question led me down a path where I noticed something:

• when we deploy a BPMN file, we transform it
• then, for the duplication check, we lookup the latest deployed version
• this either uses the cached version, or it will read it from the state and then cache it
• if it wasn't cached already but does exist in the state, it will then proceed to transform the persisted process XML
• there is no real reason why we transform it here. The latest version is only used to determine version duplicates (and in that case, return the same key, version, etc as a response)
• but this is the transformation where this specific bug happened.

So we could perhaps swat two flies at once (this is a Dutch saying):

• remove the code path that was part of this bug
• improve the performance when a new process version is deployed, when the latest isn't cached anymore (not sure if that ever happens, though).

I'm curious as to what others think about this. Is that worth it? WDYT?

//cc @remcowesterhoud @koevskinikola @berkaycanbc

[koevskinikola]

Hey @korthout, I would vote for the following:

1. Close this issue since we don't have enough data to reproduce and qualify it.
2. Create an issue for optimizing the performance of the deployment of new process versions (with a low priority).
   • I'm not sure if we should work on this soon, or until we understand the bug.

My reasoning is that currently we still don't understand why the bug is happening. The code you're suggesting for removal is just the place where the issue becomes visible (through a NoSuchElementException). By removing that code, we might miss any new occurrences of the bug, so it will become more difficult to detect and reproduce.

UPDATE: Maybe we can wrap the existing exception here into a more understandable message like: "This might indicate a bug with our deployment process, please raise an issue with our GitHub tracker"?

[korthout]

Thanks @koevskinikola

1. Agreed 👍
2. I don't think the performance improvement is a good enough reason to open an issue. The improvement could be tiny. I agree with your reasoning that "it will become more difficult to detect and reproduce" 👍

UPDATE: Maybe we can wrap the existing exception here into a more understandable message like: "This might indicate a bug with our deployment process, please raise an issue with our GitHub tracker"?

👍 I like this idea, but perhaps we can change the message. Likely, we read the error message in our own environment. So there is no need to ask users to report the bug if we already know it exists. Let's add some details about the current scenario instead.

EDIT: I've moved this back into READY, so we can make an effort to root cause this

[berkaycanbc]

Implemented a PR to create a detailed error message. We should re-open it when we encounter the log message:

Expected to find executable process in persisted process with key '%s', but after transformation no such executable process could be found

cc: @korthout