[Heartbeat] Fix fields generation #21874

andrewvc · 2020-10-16T00:48:01Z

Fields generation seems to have broken at some time in the past, but we never noticed, because we didn't add fields for quite a while. This fixes this. I had to centralize all field definitions in the top level _meta folder, but I'm fine with this change, in many ways this is actually easier.

Fields generation seems to have broken at some time in the past, but we never noticed, because we didn't add fields for quite a while. This fixes this.

elasticmachine · 2020-10-16T01:14:07Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #21874 updated
Start Time: 2020-12-10T18:08:18.810+0000
Duration: 27 min 37 sec

Test stats 🧪

Test	Results
Failed	0
Passed	932
Skipped	13
Total	945

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	932
Skipped	13
Total	945

elasticmachine · 2020-10-16T06:39:50Z

Pinging @elastic/uptime (Team:Uptime)

andrewvc · 2020-12-04T22:39:34Z

/test please

andrewkroh

Summarizing the setup here:

Only OSS heartbeat declares fields.
All fields are contained in OSS heartbeat/_meta
Each build (OSS and X-Pack) generates a fields.yml to include in packages for reference. It's not used by the beat software for anything.
There is a include/fields.go file generated from the all-in-one fields.yml file to embed the fields data into the beat.
Any fields.go files are imported somewhere to ensure they are included in the binary.

andrewkroh · 2020-12-07T15:08:17Z

heartbeat/cmd/root.go

@@ -25,12 +25,15 @@ import (
 	// include all heartbeat specific autodiscovery builders
 	_ "github.com/elastic/beats/v7/heartbeat/autodiscover/builder/hints"

-	// register default heartbeat monitors
-	_ "github.com/elastic/beats/v7/heartbeat/monitors/defaults"


This package looks broken. In the OSS magefile.go there is a generator for it. It looks like it's supposed to be creating imports for each monitor. I'd remove the package and the generation code if it's unused.

The weird thing is this used to work, and somehow broke at some point. Now, when I run mage update these lines just get deleted. That said, probably easier to just remove it and make it manual since we don't add monitor types frequently.

andrewkroh · 2020-12-07T15:12:59Z

heartbeat/scripts/mage/package.go

+	if err != nil {
+		return err
+	}
+	return devtools.GenerateAllInOneFieldsGo()


When this is run from x-pack/heartbeat it will create x-pack/heartbeat/include/fields.go, but since there are no new fields being introduced we can skip this part. We'd only want to introduce an x-pack/heartbeat/include/fields.go file if X-Pack heartbeat added new fields. And if this file was added we'd only want it to include the new fields (not any of the libbeat or OSS heartbeat fields) since they are already included on the OSS side an inherited by x-pack.

You could use some license based conditionals like in Winlogbeat

https://github.com/elastic/beats/blob/c2272c192f2a19c4b3412dfea21e25c1ca180934/winlogbeat/scripts/mage/fields.go

or do the same work in both OSS and X-Pack but write the output to OSS only

devtools.GenerateFieldsGo("fields.yml", devtools.OSSBeatDir("include/fields.go"))

It will generate new fields when the synthetics stuff is merged in #21436 , and in fact this PR is a bit of a blocker for that one. That said, if it's easier, I'd rather just include all the fields (including x-pack fields) in the OSS heartbeat and keep things simple. Does that work for you?

I went with the latter approach you suggested. Thanks for the tips!

andrewkroh · 2020-12-10T01:54:03Z

heartbeat/docs/fields.asciidoc

@@ -9570,7 +9570,7 @@ type: long

 --

-[[exported-fields-socks5]]
+[[exported-fields-socks6]]


socks6? Where did that change come from?

https://github.com/elastic/beats/pull/21874/files#diff-50eba0891861ef591b619127d66ca19a13aa297f220c1f8b44ee063144d198fcR271

Good catch, fixed!

andrewkroh · 2020-12-10T01:54:44Z

heartbeat/Makefile

@@ -13,8 +13,7 @@ collect: imports kibana
 # Generate imports for all monitors
 .PHONY: imports
 imports:


I think it's only used by heartbeat's own collect target so it can be removed if you want.

Good catch, removed

* [Heartbeat] Fix fields generation Fields generation seems to have broken at some time in the past, but we never noticed, because we didn't add fields for quite a while. This fixes this. (cherry picked from commit 5d07b87)

jsoriano · 2020-12-12T13:58:00Z

I think that this change affects the packaging build, failures like the following one can be reproduced in master now when building x-pack heartbeat packages:

Error: failed building heartbeat type=docker for platform=linux/amd64: failed to copy from fields.yml to build/package/heartbeat/heartbeat-linux-amd64.docker/docker-build/beat/fields.yml: copy failed: cannot stat source file fields.yml: stat fields.yml: no such file or directory

Seen in #22883

Fixes a bug introduced in elastic#21874 where fields.yml generation in x-pack stopped happening.

Fixes a bug introduced in #21874 where fields.yml generation in x-pack stopped happening.

Fixes a bug introduced in elastic#21874 where fields.yml generation in x-pack stopped happening. (cherry picked from commit 78a8625)

…3122) Fixes a bug introduced in #21874 where fields.yml generation in x-pack stopped happening. (cherry picked from commit 78a8625)

[Heartbeat] Fix fields generation

db8052b

Fields generation seems to have broken at some time in the past, but we never noticed, because we didn't add fields for quite a while. This fixes this.

andrewvc added the Heartbeat label Oct 16, 2020

andrewvc requested a review from andrewkroh October 16, 2020 00:48

andrewvc requested a review from a team as a code owner October 16, 2020 00:48

andrewvc self-assigned this Oct 16, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 16, 2020

andresrc added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Oct 16, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 16, 2020

andresrc requested a review from jsoriano October 16, 2020 06:39

andrewvc added 5 commits December 4, 2020 16:40

Merge remote-tracking branch 'origin/master' into fix-fields-generation

3dc2f3b

Merge remote-tracking branch 'origin/master' into fix-fields-generation

5dd9e1b

Use fields.common.yml for all field definitions

6d918d4

update x-pack fields

ab9ce14

Fix plugin loading

6aadc8f

andrewkroh reviewed Dec 7, 2020

View reviewed changes

andrewvc added 2 commits December 7, 2020 17:49

Fix defaults generation

89c1161

Merge remote-tracking branch 'origin/master' into fix-fields-generation

aa0745f

andrewvc added the [zube]: In Review label Dec 8, 2020

andrewvc added 3 commits December 9, 2020 05:42

Remove mage imports from Makefile

e0ae57a

Incorporate PR feedback

49c0501

Use proper license header for fields.go

8174d74

andrewkroh approved these changes Dec 10, 2020

View reviewed changes

andrewvc added 3 commits December 10, 2020 09:39

Incorporate PR feedback

dca409c

Mage update

03f2d6c

Remove socks6 ref

1613359

paulb-elastic added [zube]: Inbox and removed [zube]: In Review labels Dec 10, 2020

paulb-elastic added [zube]: In Progress and removed [zube]: Inbox labels Dec 10, 2020

Fix SOCKS

68f8bd5

andrewvc merged commit 5d07b87 into elastic:master Dec 10, 2020

andrewvc deleted the fix-fields-generation branch December 10, 2020 18:44

zube bot added [zube]: Done and removed [zube]: In Progress labels Dec 10, 2020

andrewvc mentioned this pull request Dec 10, 2020

Cherry-pick #21874 to 7.x: [Heartbeat] Fix fields generation #23078

Merged

andrewvc added the v7.11.0 label Dec 10, 2020

jsoriano mentioned this pull request Dec 12, 2020

Rate limit processor #22883

Merged

6 tasks

andrewvc added a commit to andrewvc/beats that referenced this pull request Dec 14, 2020

[Heartbeat] Generate fields for x-pack heartbeat as well

b4d9ebe

Fixes a bug introduced in elastic#21874 where fields.yml generation in x-pack stopped happening.

andrewvc mentioned this pull request Dec 14, 2020

[Heartbeat] Generate fields for x-pack heartbeat as well #23111

Merged

andrewvc added a commit that referenced this pull request Dec 14, 2020

[Heartbeat] Generate fields for x-pack heartbeat as well (#23111)

78a8625

Fixes a bug introduced in #21874 where fields.yml generation in x-pack stopped happening.

andrewvc mentioned this pull request Dec 14, 2020

Cherry-pick #23111 to 7.x: [Heartbeat] Generate fields for x-pack heartbeat as well #23122

Merged

andrewvc added a commit that referenced this pull request Dec 14, 2020

[Heartbeat] Generate fields for x-pack heartbeat as well (#23111) (#2…

7eb0e8c

…3122) Fixes a bug introduced in #21874 where fields.yml generation in x-pack stopped happening. (cherry picked from commit 78a8625)

paulb-elastic removed the [zube]: Done label Feb 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Heartbeat] Fix fields generation #21874

[Heartbeat] Fix fields generation #21874

andrewvc commented Oct 16, 2020 •

edited

Loading

elasticmachine commented Oct 16, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Test stats 🧪

elasticmachine commented Oct 16, 2020

andrewvc commented Dec 4, 2020

andrewkroh left a comment

andrewkroh Dec 7, 2020

andrewvc Dec 7, 2020

andrewkroh Dec 7, 2020

andrewvc Dec 7, 2020

andrewvc Dec 9, 2020

andrewkroh Dec 10, 2020

jsoriano Dec 10, 2020

andrewvc Dec 10, 2020

andrewkroh Dec 10, 2020

andrewvc Dec 10, 2020

jsoriano commented Dec 12, 2020 •

edited

Loading

[Heartbeat] Fix fields generation #21874

[Heartbeat] Fix fields generation #21874

Conversation

andrewvc commented Oct 16, 2020 • edited Loading

elasticmachine commented Oct 16, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Oct 16, 2020

andrewvc commented Dec 4, 2020

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano commented Dec 12, 2020 • edited Loading

andrewvc commented Oct 16, 2020 •

edited

Loading

elasticmachine commented Oct 16, 2020 •

edited by jenkins-beats-ci bot

Loading

jsoriano commented Dec 12, 2020 •

edited

Loading