-
Notifications
You must be signed in to change notification settings - Fork 467
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove event.original removal processors from ingest pipelines #10072
Comments
I think this is a good idea. I will occasional install a
And in order for this to work, I currently have to manually drop that |
If that change would be approved, I maybe start working on this after my current PR has been merged. |
…uth0) (#10417) Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event. This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes. auditd_manager was bumped to require 8.12.0 because the system tests were failing under 8.11.0. Relates #10072
…uth0) (elastic#10417) Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event. This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes. auditd_manager was bumped to require 8.12.0 because the system tests were failing under 8.11.0. Relates elastic#10072
) Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event. This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes. Relates #10072
…10888) Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event. This bumps the minimum version to at least 8.11.0 because it depends on the Fleet final pipeline changes. Relates #10072
Relates #7636 |
This work was partially completed through several pull requests. To finish this out, I think we should script the changes. We can review edits to each team's integrations separately to optimize the process. Basically the script needs to delete the |
…uth0) (elastic#10417) Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event. This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes. auditd_manager was bumped to require 8.12.0 because the system tests were failing under 8.11.0. Relates elastic#10072
…stic#10897) Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event. This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes. Relates elastic#10072
…lastic#10888) Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event. This bumps the minimum version to at least 8.11.0 because it depends on the Fleet final pipeline changes. Relates elastic#10072
…uth0) (elastic#10417) Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event. This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes. auditd_manager was bumped to require 8.12.0 because the system tests were failing under 8.11.0. Relates elastic#10072
…stic#10897) Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event. This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes. Relates elastic#10072
…lastic#10888) Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event. This bumps the minimum version to at least 8.11.0 because it depends on the Fleet final pipeline changes. Relates elastic#10072
It seems that most of the checked integrations have a removal processor for event.original (except if it contains the preserve tag) in their pipelines.
In the other way, the .fleet-final-pipeline also has this processor and runs after all integration pipelines (incl. the @Custom pipelines) have finished. If the integration has this removal, the @Custom pipeline cannot parse the original event anymore as it is not present in the event anymore.
I would suggest removing the removal processors from all integration ingest pipelines and forward the removal functionality into the fleet-final-pipeline where it belongs. In that case @Custom pipelines may parse the original event and can extract additional data or correct malformed parsing in the generic integrations (e.g. catalina.out in the Apache Tomcat integration):
The text was updated successfully, but these errors were encountered: