Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

m365_defender ingest pipeline failed event.original already exists #8034

Closed
ThomSwiss opened this issue Oct 2, 2023 · 2 comments · Fixed by #8090
Closed

m365_defender ingest pipeline failed event.original already exists #8034

ThomSwiss opened this issue Oct 2, 2023 · 2 comments · Fixed by #8090
Assignees
Labels
bug Something isn't working, use only for issues Integration:m365_defender Microsoft M365 Defender

Comments

@ThomSwiss
Copy link

The elastic ingest pipelines from the integration m365_defender fails, when I get the data over logstash.

Works: Elastic-Agent with m365_defender integration, Output Elasticsearch
Doesn't work: Elastic-Agent with m365_defender integration, Output Logstash. The ingest pipeline stops working with this error.message:
field [event.original] already exists

Reason: Logstash adds the field event.original which is already similar to the field message after processing with logstash.

Problem in pipeline logs-m365_defender.incident-2.2.0 :

  {
    "rename": {
      "field": "message",
      "target_field": "event.original",
      "tag": "rename_message_to_event_original",
      "ignore_missing": true
    }
  }

Possible fix:

    {
      "rename": {
        "field": "message",
        "target_field": "event.original",
        "ignore_missing": true,
        "tag": "rename_message_to_event_original",
        "ignore_failure": true
      }
    }

The problem could also exist in so some other pipeleline. Please check as well

logs-m365_defender.event
logs-m365_defender.event-2.2.0-pipeline_alert 
logs-m365_defender.event-2.2.0-pipeline_app_and_identity 
logs-m365_defender.event-2.2.0-pipeline_device 
logs-m365_defender.event-2.2.0-pipeline_email 
logs-m365_defender.incident-2.2.0 
logs-m365_defender.log-2.2.0
@bhapas
Copy link
Contributor

bhapas commented Oct 4, 2023

Relates #3451

@bhapas bhapas self-assigned this Oct 4, 2023
@bhapas bhapas added bug Something isn't working, use only for issues Team:Security-External Integrations labels Oct 4, 2023
@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working, use only for issues Integration:m365_defender Microsoft M365 Defender
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants