Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support dynamic config update #5261

Open
kkondaka opened this issue Dec 13, 2024 · 0 comments
Open

Support dynamic config update #5261

kkondaka opened this issue Dec 13, 2024 · 0 comments

Comments

@kkondaka
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
DataPrepper doesn't support changing configuration dynamically. Any changes to the config requires the data prepper to be stopped/shutdown and restarted. This is not very convenient.

Describe the solution you'd like
In most of the cases we should be able to re-initialize dataprepper components (like source, processor, sink) with new config. If a new processor is added or an existing processor is deleted, it can be deleted without having to shutdown and restart DataPrepper.
There may be still some cases where dynamic reconfiguration is not possible, in which case shutdown and restart is the only option.

Here are some simple cases where shutdown/restart of the entire pipeline may not be necessary

  1. Changing config option of a source or a sink or a processor. For example, max_retries retries options of OpenSearch Sink. We should be able change this value without having to cause any disruption to the pipeline operation
  2. Changing _when condition of a processor. Since the condition is evaluated for each event, it should be easy to replace old condition with new condition without any disruption to the pipeline operation
  3. Fixing grok pattern. This would require re-initializing grok processor object so that compileMatchPatterns is done again. This may disrupt the traffic a little because we need to pause the workers from consuming events from the buffer and then re-initialize the grok processor. This type of dynamic reconfiguration is more disruptive than the previous two examples but still much better option than shutting down the pipeline and restarting
  4. In case of push based sources, it may not be possible to support dynamic reconfiguration. For example, changing max_message_size in http source configuration might not be possible to do dynamically because it would require temporarily stopping the http source. But this may result in http client getting 4xx/5xx errors. So, in this case, only option may be to shutdown and restart the DataPrepper
  5. In case of pull based sources, it should be possible to change even source config changes as well.

Overall, the suggestion is to provide a new API to DataPrepperServer something like /update_config which sends a yaml file to the server. DataPrepper will do some validation on the yaml to see if the dynamic update can be performed. If the dynamic update cannot be performed, then rejection error should be returned. If it can be performed, then success is returned after performing the update. If some error occurs during the reconfiguration, a failure error should be returned and the pipeline config should be restored to previous configuration. It should not be partially updated.

Describe alternatives you've considered (Optional)

Additional context
Add any other context or screenshots about the feature request here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

1 participant