Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove event.original removal processors from ingest pipelines #10072

Open
Tracked by #12045
Alphayeeeet opened this issue Jun 5, 2024 · 4 comments
Open
Tracked by #12045

Remove event.original removal processors from ingest pipelines #10072

Alphayeeeet opened this issue Jun 5, 2024 · 4 comments
Labels
enhancement New feature or request Integration:All Applies to all integrations [Integration not found in source]

Comments

@Alphayeeeet
Copy link
Contributor

It seems that most of the checked integrations have a removal processor for event.original (except if it contains the preserve tag) in their pipelines.
In the other way, the .fleet-final-pipeline also has this processor and runs after all integration pipelines (incl. the @Custom pipelines) have finished. If the integration has this removal, the @Custom pipeline cannot parse the original event anymore as it is not present in the event anymore.

I would suggest removing the removal processors from all integration ingest pipelines and forward the removal functionality into the fleet-final-pipeline where it belongs. In that case @Custom pipelines may parse the original event and can extract additional data or correct malformed parsing in the generic integrations (e.g. catalina.out in the Apache Tomcat integration):

@andrewkroh
Copy link
Member

andrewkroh commented Jun 6, 2024

I think this is a good idea. I will occasional install a logs@custom pipeline that keeps the event.original when there is a pipeline failure so that I can debug it.

PUT _ingest/pipeline/logs@custom
{
  "processors": [
    {
      "append": {
        "field": "tags",
        "value": [
          "preserve_original_event"
        ],
        "allow_duplicates": false,
        "tag": "ctx.error?.message != null",
        "ignore_failure": true
      }
    }
  ]
}

And in order for this to work, I currently have to manually drop that remove processor from the managed pipelines.

@Alphayeeeet
Copy link
Contributor Author

If that change would be approved, I maybe start working on this after my current PR has been merged.

Alphayeeeet added a commit to Alphayeeeet/integrations that referenced this issue Jul 8, 2024
andrewkroh pushed a commit that referenced this issue Aug 13, 2024
…uth0) (#10417)

Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event.

This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes.

auditd_manager was bumped to require 8.12.0 because the system tests were failing under 8.11.0.

Relates #10072
jvalente-salemstate pushed a commit to jvalente-salemstate/integrations that referenced this issue Aug 21, 2024
…uth0) (elastic#10417)

Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event.

This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes.

auditd_manager was bumped to require 8.12.0 because the system tests were failing under 8.11.0.

Relates elastic#10072
andrewkroh pushed a commit that referenced this issue Sep 3, 2024
)

Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event.

This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes.

Relates #10072
andrewkroh pushed a commit that referenced this issue Sep 23, 2024
…10888)

Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event.
    
This bumps the minimum version to at least 8.11.0 because it depends on the Fleet final pipeline changes.
    
Relates #10072
@andrewkroh
Copy link
Member

Relates #7636

@andrewkroh
Copy link
Member

This work was partially completed through several pull requests. To finish this out, I think we should script the changes. We can review edits to each team's integrations separately to optimize the process. Basically the script needs to delete the remove processor for the event.original field and ensure that the integration requires at least >=8.11.0 (where the Fleet final pipeline changes were introduced).

@andrewkroh andrewkroh added Integration:All Applies to all integrations [Integration not found in source] enhancement New feature or request and removed discuss labels Dec 9, 2024
harnish-elastic pushed a commit to harnish-elastic/integrations that referenced this issue Feb 4, 2025
…uth0) (elastic#10417)

Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event.

This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes.

auditd_manager was bumped to require 8.12.0 because the system tests were failing under 8.11.0.

Relates elastic#10072
harnish-elastic pushed a commit to harnish-elastic/integrations that referenced this issue Feb 4, 2025
…stic#10897)

Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event.

This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes.

Relates elastic#10072
harnish-elastic pushed a commit to harnish-elastic/integrations that referenced this issue Feb 4, 2025
…lastic#10888)

Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event.
    
This bumps the minimum version to at least 8.11.0 because it depends on the Fleet final pipeline changes.
    
Relates elastic#10072
harnish-elastic pushed a commit to harnish-elastic/integrations that referenced this issue Feb 5, 2025
…uth0) (elastic#10417)

Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event.

This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes.

auditd_manager was bumped to require 8.12.0 because the system tests were failing under 8.11.0.

Relates elastic#10072
harnish-elastic pushed a commit to harnish-elastic/integrations that referenced this issue Feb 5, 2025
…stic#10897)

Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event.

This bumps the minimum version to 8.11.0 because it depends on the Fleet final pipeline changes.

Relates elastic#10072
harnish-elastic pushed a commit to harnish-elastic/integrations that referenced this issue Feb 5, 2025
…lastic#10888)

Delete the remove event.original processors from the pipelines because as of 8.11 the Fleet final pipeline now does this automatically when the `preserve_original_event` tag is not present in the event.
    
This bumps the minimum version to at least 8.11.0 because it depends on the Fleet final pipeline changes.
    
Relates elastic#10072
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Integration:All Applies to all integrations [Integration not found in source]
Projects
None yet
Development

No branches or pull requests

2 participants