Add event.ingested to all Filebeat modules #20386

andrewkroh · 2020-07-31T15:58:08Z

What does this PR do?

The event.ingested field defines time at which the event was ingested to Elasticsearch
and it added by the Ingest Node pipeline.

This adds a test to ensure all modules create event.ingested.

Why is it important?

This field is important when trying to build
alerts for activities that may have been reported long after they occurred (@timestamp is
much older than event.ingested). This might happen if an agent was offline for a period
of time or the processing was delayed.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Verify that no pipelines have duplicate set processors for event.ingested

Related issues

Closes [Meta] Expand support for populating event.ingested #20073

elasticmachine · 2020-07-31T16:04:51Z

Pinging @elastic/siem (Team:SIEM)

elasticmachine · 2020-07-31T16:07:59Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #20386 updated]
Start Time: 2020-08-03T20:16:22.781+0000
Duration: 54 min 30 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5516
Skipped	840
Total	6356

@timestamp

The event.ingested field defines time at which the event was ingested to Elasticsearch and it added by the Ingest Node pipeline. This field is important when trying to build alerts for activities that may have been reported long after they occurred (@timestamp is much older than event.ingested). This might happen if an agent was offline for a period of time or the processing was delayed. This adds a test to ensure all modules create event.ingested. Closes elastic#20073

andrewstucki · 2020-08-04T18:49:45Z

x-pack/filebeat/module/zeek/weird/ingest/pipeline.yml

 value: '{{_ingest.timestamp}}'
+- set:
+ field: event.created
+ value: '{{@timestamp}}'


these almost make me wonder if we should add event.created by default to the @timestamp populating code.

Zeek already had event.created, but it was being set to _ingest.timestamp so I changed it to @timestamp. The diff looks like I added a new event.created field but really I changed it.

I do think that for consistency we should add event.created and also event.original to every module. But that's probably best to do in separate pull requests.

@andrewstucki @andrewkroh
So the ECS docs have this for event.created:

In case the two timestamps are identical, @timestamp should be used.

I'm wondering if we should be setting event.created & @timestamp to the same value?

andrewstucki

I see a bunch of places where event.created is added--makes sense, but just wondering why it was added for some, but not others?

andrewkroh · 2020-08-04T19:09:32Z

Zeek already had event.created, but it was being set to _ingest.timestamp so I changed it to @timestamp. The diff looks like I added a new event.created field but really I changed it.

I do think that for consistency we should add event.created and also event.original to every module. But that's probably best to do in separate pull requests.

…ne-2.0 * upstream/master: [docs] Promote ingest management to beta (elastic#20295) Upgrade elasticsearch client library used in tests (elastic#20405) Disable logging when pulling on python integration tests (elastic#20397) Remove pillow from testing requirements.txt (elastic#20407) [Filebeat][ATP Module]Setting user agent field required by the API (elastic#20440) [Ingest Manager] Send datastreams fields (elastic#20402) Add event.ingested to all Filebeat modules (elastic#20386) [Elastic Agent] Fix agent control socket path to always be less than 107 characters (elastic#20426) Improve cgroup_regex docs with examples (elastic#20425) Makes `metrics` config option required in app_insights (elastic#20406) Ensure install scripts only install if needed (elastic#20349) Update container name for the azure filesets (elastic#19899) Group same timestamp metrics values in app_insights metricset (elastic#20403) add_process_metadata processor adds container id even if process metadata not accessible (elastic#19767) Support "cluster" scope in Metricbeat elasticsearch module (elastic#18547) [Filebeat][SophosXG Module] Renaming module and fileset (elastic#20396) Update Suricata dashboards (elastic#20394) [Elastic Agent] Improve version, restart, enroll CLI commands (elastic#20359) Prepare home directories for docker images in a different stage (elastic#20356)

…allation * upstream/master: (23 commits) [docs] Promote ingest management to beta (elastic#20295) Upgrade elasticsearch client library used in tests (elastic#20405) Disable logging when pulling on python integration tests (elastic#20397) Remove pillow from testing requirements.txt (elastic#20407) [Filebeat][ATP Module]Setting user agent field required by the API (elastic#20440) [Ingest Manager] Send datastreams fields (elastic#20402) Add event.ingested to all Filebeat modules (elastic#20386) [Elastic Agent] Fix agent control socket path to always be less than 107 characters (elastic#20426) Improve cgroup_regex docs with examples (elastic#20425) Makes `metrics` config option required in app_insights (elastic#20406) Ensure install scripts only install if needed (elastic#20349) Update container name for the azure filesets (elastic#19899) Group same timestamp metrics values in app_insights metricset (elastic#20403) add_process_metadata processor adds container id even if process metadata not accessible (elastic#19767) Support "cluster" scope in Metricbeat elasticsearch module (elastic#18547) [Filebeat][SophosXG Module] Renaming module and fileset (elastic#20396) Update Suricata dashboards (elastic#20394) [Elastic Agent] Improve version, restart, enroll CLI commands (elastic#20359) Prepare home directories for docker images in a different stage (elastic#20356) New multiline mode in Filebeat: while_pattern (elastic#19662) ...

@timestamp

The event.ingested field defines time at which the event was ingested to Elasticsearch and it added by the Ingest Node pipeline. This field is important when trying to build alerts for activities that may have been reported long after they occurred (@timestamp is much older than event.ingested). This might happen if an agent was offline for a period of time or the processing was delayed. This adds a test to ensure all modules create event.ingested. Use Filebeat read time instead of ingest time as event.created in Zeek. Closes elastic#20073 (cherry picked from commit 829c3b7)

@timestamp

The event.ingested field defines time at which the event was ingested to Elasticsearch and it added by the Ingest Node pipeline. This field is important when trying to build alerts for activities that may have been reported long after they occurred (@timestamp is much older than event.ingested). This might happen if an agent was offline for a period of time or the processing was delayed. This adds a test to ensure all modules create event.ingested. Use Filebeat read time instead of ingest time as event.created in Zeek. Closes #20073 (cherry picked from commit 829c3b7)

@timestamp

The event.ingested field defines time at which the event was ingested to Elasticsearch and it added by the Ingest Node pipeline. This field is important when trying to build alerts for activities that may have been reported long after they occurred (@timestamp is much older than event.ingested). This might happen if an agent was offline for a period of time or the processing was delayed. This adds a test to ensure all modules create event.ingested. Use Filebeat read time instead of ingest time as event.created in Zeek. Closes elastic#20073

andrewkroh added enhancement Filebeat Filebeat Team:SIEM labels Jul 31, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 31, 2020

andrewkroh force-pushed the feature/fb/event-ingested-modules branch from 5c3bb48 to ac727ec Compare July 31, 2020 16:04

andrewkroh marked this pull request as ready for review July 31, 2020 16:04

andrewkroh added the needs_backport PR is waiting to be backported to other branches. label Jul 31, 2020

andrewkroh added 2 commits August 3, 2020 15:22

Update more modules with event.ingested

6f53473

andrewkroh force-pushed the feature/fb/event-ingested-modules branch from 315e890 to 6f53473 Compare August 3, 2020 19:22

andrewkroh added 2 commits August 3, 2020 16:09

Use Filebeat read time as event.created in Zeek

6408ff3

Match panw YAML indentation (1 space)

e117b51

andrewkroh mentioned this pull request Aug 4, 2020

Add Suricata package elastic/integrations#186

Merged

2 tasks

andrewstucki reviewed Aug 4, 2020

View reviewed changes

andrewstucki approved these changes Aug 4, 2020

View reviewed changes

andrewkroh merged commit 829c3b7 into elastic:master Aug 4, 2020

andrewkroh mentioned this pull request Aug 6, 2020

Cherry-pick #20386 to 7.x: Add event.ingested to all Filebeat modules #20483

Merged

5 tasks

andrewkroh added v7.10.0 and removed needs_backport PR is waiting to be backported to other branches. labels Aug 6, 2020

adriansr mentioned this pull request Aug 20, 2020

Add event.ingested to all rsa2elk modules #20714

Merged

adriansr mentioned this pull request Aug 24, 2020

Cherry-pick #20714 to 7.x: Add event.ingested to all rsa2elk modules #20748

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add event.ingested to all Filebeat modules #20386

Add event.ingested to all Filebeat modules #20386

andrewkroh commented Jul 31, 2020 •

edited

Loading

elasticmachine commented Jul 31, 2020

elasticmachine commented Jul 31, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

andrewstucki Aug 4, 2020

leehinman Aug 10, 2020

andrewstucki left a comment

andrewkroh commented Aug 4, 2020

Add event.ingested to all Filebeat modules #20386

Add event.ingested to all Filebeat modules #20386

Conversation

andrewkroh commented Jul 31, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

Related issues

elasticmachine commented Jul 31, 2020

elasticmachine commented Jul 31, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

andrewstucki Aug 4, 2020

Choose a reason for hiding this comment

leehinman Aug 10, 2020

Choose a reason for hiding this comment

andrewstucki left a comment

Choose a reason for hiding this comment

andrewkroh commented Aug 4, 2020

andrewkroh commented Jul 31, 2020 •

edited

Loading

elasticmachine commented Jul 31, 2020 •

edited by jenkins-beats-ci bot

Loading