Cherry-pick #23240 to 7.12: libbeat/template: duplicate entries in fields.yml leads to repeated dynamic templates #24197

axw · 2021-02-24T07:10:46Z

Cherry-pick of PR #23240 to 7.12 branch. Original message:

What does this PR do?

In case of duplicate fields, do not generate duplicate dynamic_template items. We already deduplicate field mappings because there we update a map.

There's some light refactoring here to stop using a global list of dynamic templates, and instead maintain a map of dynamic templates on the processor.

Why is it important?

As part of APM Server's migration to Fleet and data streams, we now duplicate fields for each data stream. We still need to generate legacy templates for 7.x, so we gather all of these data stream fields.yml files together to generate docs, templates, etc. The combined fields.yml will therefore have duplicate entries.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~- [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

How to test this PR locally

N/A

Related issues

elastic/apm-server#4576

…ynamic templates (elastic#23240) * libbeat/template: deduplicate dynamic templates In case of duplicate fields, do not generate duplicate dynamic_template items. We already deduplicate field mappings because we update a map. * libbeat/template: preserve dynamic template order (cherry picked from commit c0bfea4)

elasticmachine · 2021-02-24T07:10:48Z

Pinging @elastic/integrations-services (Team:Services)

elasticmachine · 2021-02-24T08:35:23Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #24197 opened
Start Time: 2021-02-24T07:13:01.827+0000
Duration: 82 min 47 sec
Commit: 0052e3e

Test stats 🧪

Test	Results
Failed	1
Passed	46608
Skipped	4975
Total	51584

Trends 🧪

Test errors

Expand to view the tests failures

`Build&Test / auditbeat-windows-8-windows-8 / TestNonRecursive – github.com/elastic/beats/v7/auditbeat/module/file_integrity/monitor`

Expand to view the error details

 Failed

Expand to view the stacktrace

 === RUN   TestNonRecursive
    monitor_test.go:73: 
        	Error Trace:	monitor_test.go:73
        	Error:      	An error is expected but got nil.
        	Test:       	TestNonRecursive
    monitor_test.go:74: 
        	Error Trace:	monitor_test.go:74
        	Error:      	Not equal: 
        	            	expected: *errors.errorString(&errors.errorString{s:"read timeout"})
        	            	actual  : <nil>(<nil>)
        	Test:       	TestNonRecursive
--- FAIL: TestNonRecursive (4.79s)

Steps errors

Expand to view the steps failures

`auditbeat-windows-8-windows-8 - mage build unitTest`

Took 3 min 12 sec . View more details on here
Description: mage build unitTest

`gsutil -m -q cp -a public-read test-build-artifacts-auditbeat-windows-8-windows-8.tgz gs://beats-ci-`

Took 0 min 4 sec . View more details on here
Description: @echo off gsutil -m -q cp -a public-read test-build-artifacts-auditbeat-windows-8-windows-8.tgz gs://beats-ci-temp/Beats/beats/PR-24197-1

`Error signal`

Took 0 min 2 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-02-24T08:34:19.067Z] 36.29s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_171_checkpoint
[2021-02-24T08:34:19.067Z] 21.30s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_000_suricata
[2021-02-24T08:34:19.067Z] 15.26s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_140_snort
[2021-02-24T08:34:19.067Z] 14.70s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_110_o365
[2021-02-24T08:34:19.067Z] 13.13s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_212_threatintel
[2021-02-24T08:34:19.067Z] 12.25s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_109_o365
[2021-02-24T08:34:19.067Z] 9.41s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_178_cisco
[2021-02-24T08:34:19.067Z] 9.28s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_242_ibmmq
[2021-02-24T08:34:19.067Z] 9.10s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_213_threatintel
[2021-02-24T08:34:19.067Z] 9.00s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_226_panw
[2021-02-24T08:34:19.067Z] 8.87s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_186_cisco
[2021-02-24T08:34:19.067Z] 8.87s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_196_cisco
[2021-02-24T08:34:19.067Z] 8.78s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_230_panw
[2021-02-24T08:34:19.067Z] 8.66s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_229_panw
[2021-02-24T08:34:19.067Z] 8.61s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_243_ibmmq
[2021-02-24T08:34:19.067Z] 8.29s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_255_oracle
[2021-02-24T08:34:19.067Z] 8.19s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_141_juniper
[2021-02-24T08:34:19.067Z] 8.13s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_270_gcp
[2021-02-24T08:34:19.067Z] 8.08s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_117_zscaler
[2021-02-24T08:34:19.067Z] 8.07s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_173_squid
[2021-02-24T08:34:19.067Z] ================ 317 passed, 304 warnings in 1359.15s (0:22:39) ================
[2021-02-24T08:34:19.067Z] >> python test: Integration Testing Complete
[2021-02-24T08:34:22.891Z] Client: Docker Engine - Community
[2021-02-24T08:34:22.891Z]  Version:           20.10.3
[2021-02-24T08:34:22.891Z]  API version:       1.41
[2021-02-24T08:34:22.891Z]  Go version:        go1.13.15
[2021-02-24T08:34:22.891Z]  Git commit:        48d30b5
[2021-02-24T08:34:22.891Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-02-24T08:34:22.891Z]  OS/Arch:           linux/amd64
[2021-02-24T08:34:22.891Z]  Context:           default
[2021-02-24T08:34:22.891Z]  Experimental:      true
[2021-02-24T08:34:22.891Z] 
[2021-02-24T08:34:22.891Z] Server: Docker Engine - Community
[2021-02-24T08:34:22.891Z]  Engine:
[2021-02-24T08:34:22.891Z]   Version:          20.10.3
[2021-02-24T08:34:22.891Z]   API version:      1.41 (minimum version 1.12)
[2021-02-24T08:34:22.891Z]   Go version:       go1.13.15
[2021-02-24T08:34:22.891Z]   Git commit:       46229ca
[2021-02-24T08:34:22.891Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-02-24T08:34:22.891Z]   OS/Arch:          linux/amd64
[2021-02-24T08:34:22.891Z]   Experimental:     false
[2021-02-24T08:34:22.891Z]  containerd:
[2021-02-24T08:34:22.891Z]   Version:          1.4.3
[2021-02-24T08:34:22.891Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-02-24T08:34:22.891Z]  runc:
[2021-02-24T08:34:22.891Z]   Version:          1.0.0-rc92
[2021-02-24T08:34:22.891Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-02-24T08:34:22.891Z]  docker-init:
[2021-02-24T08:34:22.891Z]   Version:          0.19.0
[2021-02-24T08:34:22.891Z]   GitCommit:        de40ad0
[2021-02-24T08:34:22.891Z] Unable to find image 'alpine:3.4' locally
[2021-02-24T08:34:23.460Z] 3.4: Pulling from library/alpine
[2021-02-24T08:34:23.720Z] c1e54eec4b57: Pulling fs layer
[2021-02-24T08:34:23.981Z] c1e54eec4b57: Download complete
[2021-02-24T08:34:24.241Z] c1e54eec4b57: Pull complete
[2021-02-24T08:34:24.241Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-02-24T08:34:24.241Z] Status: Downloaded newer image for alpine:3.4
[2021-02-24T08:34:26.471Z] + python .ci/scripts/pre_archive_test.py
[2021-02-24T08:34:28.383Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-02-24T08:34:28.437Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24197/src/github.com/elastic/beats/build
[2021-02-24T08:34:28.816Z] + rm -rf ve
[2021-02-24T08:34:28.816Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-02-24T08:34:28.864Z] Recording test results
[2021-02-24T08:34:31.831Z] [Checks API] No suitable checks publisher found.
[2021-02-24T08:34:32.427Z] Client: Docker Engine - Community
[2021-02-24T08:34:32.427Z]  Version:           20.10.3
[2021-02-24T08:34:32.427Z]  API version:       1.41
[2021-02-24T08:34:32.427Z]  Go version:        go1.13.15
[2021-02-24T08:34:32.427Z]  Git commit:        48d30b5
[2021-02-24T08:34:32.427Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-02-24T08:34:32.427Z]  OS/Arch:           linux/amd64
[2021-02-24T08:34:32.427Z]  Context:           default
[2021-02-24T08:34:32.427Z]  Experimental:      true
[2021-02-24T08:34:32.427Z] 
[2021-02-24T08:34:32.427Z] Server: Docker Engine - Community
[2021-02-24T08:34:32.427Z]  Engine:
[2021-02-24T08:34:32.427Z]   Version:          20.10.3
[2021-02-24T08:34:32.427Z]   API version:      1.41 (minimum version 1.12)
[2021-02-24T08:34:32.427Z]   Go version:       go1.13.15
[2021-02-24T08:34:32.427Z]   Git commit:       46229ca
[2021-02-24T08:34:32.427Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-02-24T08:34:32.427Z]   OS/Arch:          linux/amd64
[2021-02-24T08:34:32.427Z]   Experimental:     false
[2021-02-24T08:34:32.427Z]  containerd:
[2021-02-24T08:34:32.427Z]   Version:          1.4.3
[2021-02-24T08:34:32.427Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-02-24T08:34:32.427Z]  runc:
[2021-02-24T08:34:32.427Z]   Version:          1.0.0-rc92
[2021-02-24T08:34:32.427Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-02-24T08:34:32.427Z]  docker-init:
[2021-02-24T08:34:32.427Z]   Version:          0.19.0
[2021-02-24T08:34:32.427Z]   GitCommit:        de40ad0
[2021-02-24T08:34:47.249Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-02-24T08:34:47.373Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24197/src/github.com/elastic/beats
[2021-02-24T08:34:48.243Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-24197
[2021-02-24T08:34:48.341Z] [INFO] getVaultSecret: Getting secrets
[2021-02-24T08:34:48.430Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-02-24T08:34:49.439Z] + chmod 755 generate-build-data.sh
[2021-02-24T08:34:49.439Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24197/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24197/runs/1 FAILURE 4907338
[2021-02-24T08:34:49.690Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24197/runs/1/steps/?limit=10000 -o steps-info.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	1
Passed	46608
Skipped	4975
Total	51584

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / auditbeat-windows-8-windows-8 / TestNonRecursive – github.com/elastic/beats/v7/auditbeat/module/file_integrity/monitor

axw added [zube]: In Review backport Team:Services (Deprecated) Label for the former Integrations-Services team labels Feb 24, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 24, 2021

axw requested a review from simitt February 24, 2021 07:10

simitt approved these changes Feb 24, 2021

View reviewed changes

axw merged commit 2ff2bd1 into elastic:7.12 Feb 24, 2021

axw deleted the backport_23240_7.12 branch February 24, 2021 10:37

zube bot added [zube]: Done and removed [zube]: In Review labels Feb 24, 2021

zube bot removed the [zube]: Done label May 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick #23240 to 7.12: libbeat/template: duplicate entries in fields.yml leads to repeated dynamic templates #24197

Cherry-pick #23240 to 7.12: libbeat/template: duplicate entries in fields.yml leads to repeated dynamic templates #24197

axw commented Feb 24, 2021 •

edited by zube bot

Loading

elasticmachine commented Feb 24, 2021

elasticmachine commented Feb 24, 2021

Build stats

Test stats 🧪

Trends 🧪

`Build&Test / auditbeat-windows-8-windows-8 / TestNonRecursive – github.com/elastic/beats/v7/auditbeat/module/file_integrity/monitor`

`auditbeat-windows-8-windows-8 - mage build unitTest`

`gsutil -m -q cp -a public-read test-build-artifacts-auditbeat-windows-8-windows-8.tgz gs://beats-ci-`

`Error signal`

Test stats 🧪

Genuine test errors

Cherry-pick #23240 to 7.12: libbeat/template: duplicate entries in fields.yml leads to repeated dynamic templates #24197

Cherry-pick #23240 to 7.12: libbeat/template: duplicate entries in fields.yml leads to repeated dynamic templates #24197

Conversation

axw commented Feb 24, 2021 • edited by zube bot Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Feb 24, 2021

elasticmachine commented Feb 24, 2021

💔 Tests Failed

Build stats

Test stats 🧪

Trends 🧪

Test errors

Build&Test / auditbeat-windows-8-windows-8 / TestNonRecursive – github.com/elastic/beats/v7/auditbeat/module/file_integrity/monitor

Steps errors

auditbeat-windows-8-windows-8 - mage build unitTest

gsutil -m -q cp -a public-read test-build-artifacts-auditbeat-windows-8-windows-8.tgz gs://beats-ci-

Error signal

Log output

🐛 Flaky test report

Test stats 🧪

Genuine test errors

axw commented Feb 24, 2021 •

edited by zube bot

Loading

`Build&Test / auditbeat-windows-8-windows-8 / TestNonRecursive – github.com/elastic/beats/v7/auditbeat/module/file_integrity/monitor`

`auditbeat-windows-8-windows-8 - mage build unitTest`

`gsutil -m -q cp -a public-read test-build-artifacts-auditbeat-windows-8-windows-8.tgz gs://beats-ci-`

`Error signal`