Update to ECS 8.0 #28620

adriansr · 2021-10-25T13:28:03Z

What does this PR do?

Updates the ECS field definitions to the current master (8.0-dev).

Why is it important?

To use the latest version of ECS.

Breaking changes

Remove log.original (@adriansr)
Remove process.ppid (@adriansr)
Remove host.user.* field reuse (@adriansr)
Don't lowercase http.request.method (@adriansr)
~~- [ ] Remove deprecation notice from http.request.method (@marc-gr)~~
~~- [ ] Migrate log.origin.file.line field type from integer to long (@marc-gr)~~
Remove Go code generator (@marc-gr)

Added

~~- [ ] Function as a Service fields (Beta) (@marc-gr)~~

Improvements

~~- [ ] Wildcard type field migration stage 3 (GA)~~
~~- [ ] match_only_text field type migration stage 3 (GA)~~
~~- [ ] Threat indicator fields stage 3 (GA) (@marc-gr)~~ * Changes to TI fields are on hold, as Beats don't use the threat.* fields from ECS.

Note to reviewers

All liblogparser.js files include the same changes (same file):

37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/cylance/protect/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/zscaler/zia/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/tomcat/log/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/squid/log/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/microsoft/dhcp/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/f5/bigipafm/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/f5/bigipapm/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/proofpoint/emailsecurity/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/barracuda/spamfirewall/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/barracuda/waf/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/netscout/sightline/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/bluecoat/director/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/cisco/nexus/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/cisco/meraki/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/sonicwall/firewall/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/radware/defensepro/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/snort/log/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/infoblox/nios/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/juniper/netscreen/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/juniper/junos/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/sophos/utm/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/imperva/securesphere/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/fortinet/fortimail/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/fortinet/fortimanager/config/liblogparser.js
37fb671452818e5ef3d6f03810fd904cdacb9d8b  ./module/fortinet/clientendpoint/config/liblogparser.js

elasticmachine · 2021-10-25T15:42:30Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-11-03T22:45:24.771+0000
Duration: 232 min 23 sec
Commit: 98254b9

Test stats 🧪

Test	Results
Failed	0
Passed	53596
Skipped	5242
Total	58838

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

adriansr · 2021-10-29T09:56:39Z

/test

elasticmachine · 2021-10-29T15:58:11Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-10-29T15:58:13Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

mergify · 2021-11-01T13:10:50Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b ecs_8.0 upstream/ecs_8.0
git merge upstream/master
git push upstream ecs_8.0

P1llus

Nothing standing out from my point. Checked that each beat has the updated versions applied, checked that the ingest pipelines included in the commit is updated properly, and the expected output looks good.
Did not pull it down to check if there was any more files that should have been added. If we feel that is required I can do that as well.

Just some small comments from my side, then LGTM.

P1llus · 2021-11-02T11:00:06Z

NOTICE.txt

--------------------------------------------------------------------------------
-Dependency : github.com/elastic/ecs


Is this part totally removed now, or is it just github that messes it up?

Yes, it's removed because it's not available anymore in the ECS repo. We could have left the dependency pointing to v1.12 but eventually they could diverge, forcing us to fork anyway.

P1llus · 2021-11-02T11:08:29Z

go.mod

@@ -62,7 +62,6 @@ require (
 	github.com/dustin/go-humanize v1.0.0
 	github.com/eapache/go-resiliency v1.2.0
 	github.com/eclipse/paho.mqtt.golang v1.3.5
-	github.com/elastic/ecs v1.12.0


Are we completely removing it as a dependency? From my understanding I think we are, just making sure.

mergify · 2021-11-02T12:17:00Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b ecs_8.0 upstream/ecs_8.0
git merge upstream/master
git push upstream ecs_8.0

Replace process.ppid with process.parent.pid

This is a better mapping for the `msg` rsa2elk field that was mapped to log.original.

adriansr · 2021-11-04T16:25:02Z

@Mergifyio refresh

Updates the ECS field definitions to the current master (8.0-dev). - add_process_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. - add_docker_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. - Auditbeat: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. - Auditbeat system/process metricset: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. - Filebeat all modules: Replace usages of deprecated ECS fields `process.ppid` and `log.original` with `process.parent.pid` and `event.original`. - Filebeat: Replace usages of `host.user.*` fields with `user.*` in `cisco`, `microsoft` and `oracle` modules. - Packetbeat http: The field `http.request.method` will maintain its original case. Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> (cherry picked from commit dde7a1f) # Conflicts: # x-pack/filebeat/module/cisco/ios/test/cisco-ios-syslog.log-expected.json # x-pack/filebeat/module/iptables/log/test/geo.log-expected.json # x-pack/filebeat/module/panw/panos/test/global_protect.log-expected.json # x-pack/filebeat/module/panw/panos/test/hipmatch.log-expected.json # x-pack/filebeat/module/panw/panos/test/pan_inc_other.log-expected.json # x-pack/filebeat/module/panw/panos/test/pan_inc_threat.log-expected.json # x-pack/filebeat/module/panw/panos/test/pan_inc_traffic.log-expected.json # x-pack/filebeat/module/panw/panos/test/pan_inc_traffic_ietf.log-expected.json # x-pack/filebeat/module/panw/panos/test/traffic.log-expected.json # x-pack/filebeat/module/sonicwall/firewall/test/general.log-expected.json # x-pack/filebeat/module/sophos/utm/test/generated.log-expected.json # x-pack/filebeat/module/squid/log/test/access1.log-expected.json

mergify · 2021-11-04T16:25:42Z

refresh

✅ Pull request refreshed

Updates the ECS field definitions to the current master (8.0-dev). - add_process_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. - add_docker_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. - Auditbeat: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. - Auditbeat system/process metricset: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. - Filebeat all modules: Replace usages of deprecated ECS fields `process.ppid` and `log.original` with `process.parent.pid` and `event.original`. - Filebeat: Replace usages of `host.user.*` fields with `user.*` in `cisco`, `microsoft` and `oracle` modules. - Packetbeat http: The field `http.request.method` will maintain its original case. (cherry picked from commit dde7a1f) Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> Co-authored-by: Adrian Serrano <adrisr83@gmail.com>

adriansr added enhancement in progress Pull request is currently in progress. Team:Integrations Label for the Integrations team Team:Security-External Integrations 8.0-candidate labels Oct 25, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 25, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 25, 2021

elastic deleted a comment from mergify bot Oct 25, 2021

adriansr mentioned this pull request Oct 25, 2021

[breaking] Make default_field: false the default for all fields #28596

Merged

3 tasks

adriansr force-pushed the ecs_8.0 branch from 119de05 to 072736c Compare October 29, 2021 14:20

adriansr marked this pull request as ready for review October 29, 2021 15:58

adriansr requested review from a team as code owners October 29, 2021 15:58

adriansr added review and removed in progress Pull request is currently in progress. labels Oct 29, 2021

adriansr changed the title ~~[Draft] Update ECS field definitions to 8.0~~ Update to ECS 8.0 Oct 29, 2021

adriansr force-pushed the ecs_8.0 branch from 9bf5396 to 904d8ef Compare November 2, 2021 10:45

P1llus approved these changes Nov 2, 2021

View reviewed changes

adriansr added 5 commits November 3, 2021 12:15

Update fields.ecs.yml to ECS 8.0

5307e9d

Make update

b6cebf3

Auditbeat: use parent.pid

e502d8e

Replace process.ppid with process.parent.pid

Metricbeat

4fc09d3

add_process_metadata

3af5143

adriansr added 16 commits November 3, 2021 12:15

rsa2elk: Use message instead of log.original

68a744e

This is a better mapping for the `msg` rsa2elk field that was mapped to log.original.

Zookeeper: Use event.original instead of log.original

d51a007

iptables: Use event.original instead of log.original

e8898ac

mssql: Use event.original instead of log.original

bd3dbe7

microsoft: remove host.user

5c177f2

oracle: remove host.user

51e1ebd

cisco: remove host.user

8dcae91

Update NOTICE

db9b4fe

filebeat auditd: replace ppid usage

3a3390f

filebeat auditd: get rid of ppid

f8102cc

Fix add_docker_metadata sample config

2f70f7b

Missing update

9c88964

Packetbeat: Don't lowercase http request method

c424368

cisco/ios: Fix test

f38ef20

Packetbeat: Fix system test

56a3735

Update changelog

1dcd370

adriansr force-pushed the ecs_8.0 branch from 904d8ef to 1dcd370 Compare November 3, 2021 11:29

adriansr added 2 commits November 3, 2021 16:44

Re-generate expected files

2da602b

Update to latest ECS 8.0-dev

98254b9

adriansr merged commit dde7a1f into elastic:master Nov 4, 2021

adriansr added backport-v8.0.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Nov 4, 2021

mergify bot mentioned this pull request Nov 4, 2021

[8.0](backport #28620) Update to ECS 8.0 #28815

Merged

andrewkroh mentioned this pull request Dec 2, 2021

docs: Prepare Changelog for 7.16.0 #29245

Merged

This was referenced Jan 27, 2022

[libbeat] add_process_metadata - add process.parent.pid in 7.x #30070

Closed

ECS 8.0 Meta Issue elastic/integrations#2363

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to ECS 8.0 #28620

Update to ECS 8.0 #28620

adriansr commented Oct 25, 2021 •

edited

Loading

elasticmachine commented Oct 25, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

adriansr commented Oct 29, 2021

elasticmachine commented Oct 29, 2021

elasticmachine commented Oct 29, 2021

mergify bot commented Nov 1, 2021

P1llus left a comment

P1llus Nov 2, 2021

adriansr Nov 3, 2021

P1llus Nov 2, 2021

mergify bot commented Nov 2, 2021

adriansr commented Nov 4, 2021

mergify bot commented Nov 4, 2021

		--------------------------------------------------------------------------------
		Dependency : github.com/elastic/ecs

Update to ECS 8.0 #28620

Update to ECS 8.0 #28620

Conversation

adriansr commented Oct 25, 2021 • edited Loading

What does this PR do?

Why is it important?

Breaking changes

Added

Improvements

Note to reviewers

elasticmachine commented Oct 25, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

adriansr commented Oct 29, 2021

elasticmachine commented Oct 29, 2021

elasticmachine commented Oct 29, 2021

mergify bot commented Nov 1, 2021

P1llus left a comment

Choose a reason for hiding this comment

P1llus Nov 2, 2021

Choose a reason for hiding this comment

adriansr Nov 3, 2021

Choose a reason for hiding this comment

P1llus Nov 2, 2021

Choose a reason for hiding this comment

mergify bot commented Nov 2, 2021

adriansr commented Nov 4, 2021

mergify bot commented Nov 4, 2021

✅ Pull request refreshed

adriansr commented Oct 25, 2021 •

edited

Loading

elasticmachine commented Oct 25, 2021 •

edited by jenkins-beats-ci bot

Loading