-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MINOR: Improve PeriodicBatchEntityTrigger Workflows performance #18664
Conversation
Quality Gate passed for 'open-metadata-ingestion'Issues Measures |
The test that is failing for the CI is being fixed here: https://github.com/open-metadata/OpenMetadata/pull/18640/files |
@@ -40,6 +40,10 @@ private WorkflowHandler(OpenMetadataApplicationConfig config) { | |||
ProcessEngineConfiguration processEngineConfiguration = | |||
new StandaloneProcessEngineConfiguration() | |||
.setAsyncExecutorActivate(true) | |||
.setAsyncExecutorCorePoolSize(50) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should these be configurable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think they should, but we can iterate on that after having a decent default for production load.
I was thinking about having a Workflows Setting
kind of thing within the OpenMetadata settings.
What do you think?
* Update default configurations * Refactor how we call the main Workflow in order to increase performance
In order to Increase the performance of the Flowable Workflows (Mainly for the PeriodicBatchEntityTrigger), we are:
Updating the default Async Executor Flowable configurations to handle more workload
The default configurations are pretty bad for an asset intensive production environment and thus we are updating them.
Refactoring how we call the MainWorkflow from the TriggerWorkflow
We are now relying on
CallActivity
to call the MainWorkflow instead of doing it programatically through aJavaDelegate
on aServiceTask
.This is pretty important for the PeriodicBatchEntityTrigger, where we are now relying on
MultiInstanceLoopCharacteristics
for the concurrent task execution instead of dealing with it manually.It is a BPMN2.0 natural way of solving this issues that I didn't realized it would work for a whole workflow before.
As a plus, it also takes care of waiting for the MainWorkflow to finish before finishing the TriggerWorkflow.
Small fix to avoid patching an Entity if the updated Certification would be the same as the one they currently have.
Type of change:
Checklist:
Fixes <issue-number>: <short explanation>