Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add: Error Handling Environment Bundle to Bento configuration #174

Merged
merged 8 commits into from
Dec 6, 2024

Conversation

gregfurman
Copy link
Collaborator

@gregfurman gregfurman commented Nov 23, 2024

Motivation

We need global config to allow for a processing step to return and log an error, with the subsequent processors not being run.

Related to #166 -- where we opted for nacking at the failed processor instead of at the output layer.

Can be used as follows:

pipeline:
  processors:
    - mapping: |
        root = throw(this.value)
    - mapping: |
        root.message = "I'm never reached..."

error_handling:
  strategy: reject

Changes

  • Creates config for a new error_handling in the top-level that can be one of reject or none
  • Adds an interface that iterates through all messages from a processor and checks if any contain an error.
  • Failures will be nacked at the input layer without needing to reach the output.
  • Fixes the generate input component always enabling auto_replay_nacks.

TODO

  • Add more tests.
  • Populate this config at the manager level to allow for nested input.processors and output.processors to also have their results checked for errors.
  • Look into adding backoff for retries and shutting off the input to stop new data being read in.
  • Add documentation for this feature.

@gregfurman gregfurman self-assigned this Nov 23, 2024
@gregfurman gregfurman marked this pull request as ready for review November 24, 2024 17:23
Copy link
Collaborator

@jem-davies jem-davies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just change the key strict_mode to something like on_error with the values being an enum of, continue (default) / strict_mode.

So that we can add new modes later on if required, or to add a new mode to change the retry behaviour later? i.e. a new on_error: exponential_backoff.

I think that adding this field to the manager level also makes sense when running in streams mode, it will be ever stream that is created will have the same strict_mode / on_error mode - will simplify understanding?

@gregfurman gregfurman changed the title add: Strict Mode option to Pipeline configuration add: Error Handling Environment Bundle to Bento configuration Dec 1, 2024
Co-authored-by: Jem Davies <131159520+jem-davies@users.noreply.github.com>
@gregfurman gregfurman merged commit cfeec19 into warpstreamlabs:main Dec 6, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants