Regression in deploying large payloads

[korthout]

We've introduced a regression on deploying large payloads on multi-partition clusters with

• Switch Deployment:Create processor to new record distribution behavior #11661

Before that pull request, a deployed resource was written in two follow-up events:

• Deployment:CREATED
• Process:CREATED (or DecisionRequirements:CREATED depending on resource)

The specialized DeploymentDistribution would take the resource from the Deployment:CREATED to distribute it to the other partitions.

With #11456, a new event CommandDistribution:STARTED was introduced to store the command for distribution. For deployments, this contains the entire Deployment incl. the resource. This event is only appended on multi-partition clusters. But, when it is appended, it further reduces the maximum payload size because the follow-up events reach the MAX_BATCH_SIZE restriction with a lower payload size.

Note The MAX_BATCH_SIZE is a limitation that originated from the MAX_MESSAGE_SIZE configuration setting but nowadays is only defined by the LogStreamBuilder's MAX_FRAGMENT_SIZE: 4MB.

This regression lowers the maximum payload size of deployments from ~2MB down to ~1.4MB.

The regression does not exist on single partition clusters as these do not require the distribution of the deployment.

---

Suggested solution:

• don't write the resource in the Deployment:CREATED event, only in the Process:CREATED and the DecisionRequirements:CREATED events (estimate: x-small)

Of course, we'd need to inform users that we're no longer writing the resource for that event in the Update Guide. As this would be breaking user space. The documentation should clarify that the resource is available in the Process:CREATED and the DecisionRequirements:CREATED instead.

[korthout]

There are several solution ideas:

• Allow writing arbitrarily large Raft entries/batches #11513
  • Does not seem trivial to implement
  • Mostly a ZDP effort
• don't write the resource in the Deployment:CREATED event, only in the Process:CREATED and the DecisionRequirements:CREATED events (estimate: x-small)
  • would require alignment with Operate and Optimize to make sure they consume the resource from Process:CREATED

[korthout]

@sdorokhova Can you verify for me that Operate consumes the Process:CREATED event to get the BPMN model and not the Deployment:CREATED event? Likewise, does it consume the DecisionRequirements:CREATED event for DMN models?

@RomanJRW Can you verify the same as above, but for Optimize?

[sdorokhova]

Hi @korthout ,
we read processes from process index and decision requirements from decision-requirements index. You can check here all the indices we're reading from. Is this what you were asking about?

[korthout]

@sdorokhova Perfect! That's exactly what I mean. I should've mentioned that it concerns the ES indices zeebe-record-process and zeebe-record-decision-requirements.

I was hoping you weren't reading the BPMN/DMN resources from the zeebe-record-deployment index.

[megglos]

@korthout do I understand this correctly that this regression is only present in 8.3? And the changes that happened in 8.2 like #11660 have not caused a regression?

[korthout]

@megglos Correct, the regression only exists on main and if left unfixed will only be present in the upcoming 8.3. The reason is that deployment distribution has been switched over only for 8.3(-alpha*), not for 8.2.

[RomanJRW]

Hey @korthout - apologies for slow response, I have had FTO. I can confirm the same for Optimize, that we read from zeebe-record-process and not the deployment index

[korthout]

Now that both Operate and Optimize have confirmed that they don't access the resource from the zeebe-record-deployment index, I think it's fine to take this solution:

• don't write the resource in the Deployment:CREATED event, only in the Process:CREATED and the DecisionRequirements:CREATED events (estimate: x-small)

Of course, we'd need to inform users that we're no longer writing the resource for that event in the Update Guide. As this would be breaking user space. The documentation should clarify that the resource is available in the Process:CREATED and the DecisionRequirements:CREATED instead.