You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
DataPrepper doesn't support changing configuration dynamically. Any changes to the config requires the data prepper to be stopped/shutdown and restarted. This is not very convenient.
Describe the solution you'd like
In most of the cases we should be able to re-initialize dataprepper components (like source, processor, sink) with new config. If a new processor is added or an existing processor is deleted, it can be deleted without having to shutdown and restart DataPrepper.
There may be still some cases where dynamic reconfiguration is not possible, in which case shutdown and restart is the only option.
Here are some simple cases where shutdown/restart of the entire pipeline may not be necessary
Changing config option of a source or a sink or a processor. For example, max_retries retries options of OpenSearch Sink. We should be able change this value without having to cause any disruption to the pipeline operation
Changing _when condition of a processor. Since the condition is evaluated for each event, it should be easy to replace old condition with new condition without any disruption to the pipeline operation
Fixing grok pattern. This would require re-initializing grok processor object so that compileMatchPatterns is done again. This may disrupt the traffic a little because we need to pause the workers from consuming events from the buffer and then re-initialize the grok processor. This type of dynamic reconfiguration is more disruptive than the previous two examples but still much better option than shutting down the pipeline and restarting
In case of push based sources, it may not be possible to support dynamic reconfiguration. For example, changing max_message_size in http source configuration might not be possible to do dynamically because it would require temporarily stopping the http source. But this may result in http client getting 4xx/5xx errors. So, in this case, only option may be to shutdown and restart the DataPrepper
In case of pull based sources, it should be possible to change even source config changes as well.
Overall, the suggestion is to provide a new API to DataPrepperServer something like /update_config which sends a yaml file to the server. DataPrepper will do some validation on the yaml to see if the dynamic update can be performed. If the dynamic update cannot be performed, then rejection error should be returned. If it can be performed, then success is returned after performing the update. If some error occurs during the reconfiguration, a failure error should be returned and the pipeline config should be restored to previous configuration. It should not be partially updated.
Describe alternatives you've considered (Optional)
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
DataPrepper doesn't support changing configuration dynamically. Any changes to the config requires the data prepper to be stopped/shutdown and restarted. This is not very convenient.
Describe the solution you'd like
In most of the cases we should be able to re-initialize dataprepper components (like source, processor, sink) with new config. If a new processor is added or an existing processor is deleted, it can be deleted without having to shutdown and restart DataPrepper.
There may be still some cases where dynamic reconfiguration is not possible, in which case shutdown and restart is the only option.
Here are some simple cases where shutdown/restart of the entire pipeline may not be necessary
max_retries
retries options of OpenSearch Sink. We should be able change this value without having to cause any disruption to the pipeline operation_when
condition of a processor. Since the condition is evaluated for each event, it should be easy to replace old condition with new condition without any disruption to the pipeline operationcompileMatchPatterns
is done again. This may disrupt the traffic a little because we need to pause the workers from consuming events from the buffer and then re-initialize the grok processor. This type of dynamic reconfiguration is more disruptive than the previous two examples but still much better option than shutting down the pipeline and restartingmax_message_size
in http source configuration might not be possible to do dynamically because it would require temporarily stopping the http source. But this may result in http client getting 4xx/5xx errors. So, in this case, only option may be to shutdown and restart the DataPrepperOverall, the suggestion is to provide a new API to DataPrepperServer something like
/update_config
which sends a yaml file to the server. DataPrepper will do some validation on the yaml to see if the dynamic update can be performed. If the dynamic update cannot be performed, then rejection error should be returned. If it can be performed, then success is returned after performing the update. If some error occurs during the reconfiguration, a failure error should be returned and the pipeline config should be restored to previous configuration. It should not be partially updated.Describe alternatives you've considered (Optional)
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: