Migration of metricset.* fields to ECS #8941

ruflin · 2018-11-06T10:30:00Z

Add event.dataset as {module}.{metricset} but keep metricset.name around for backward compatiblity.
Add event.duration which comes from metricset.rtt but is now in nano instead of micro seconds. event.duration will be introduce in 6.x already.
Rename metricset.module to event.module
Rename metricset.host to service.address
Removed metricset.namespace as this becomes event.dataset and should now always be set.

Further changes

Updated some of the data.json files as example
Migration file updated
Old field definitions removed
System tests updated

ruflin · 2018-11-06T10:30:34Z

metricbeat/mb/event.go

 	}
 	if event.Host != "" {
-		info["host"] = event.Host
+		e.Put("service.hostname", event.Host)


@webmat To make things a bit more interesting, Host here contains also the Port number. For example elasticsearch:9200.

andrewkroh · 2018-11-06T14:34:45Z

metricbeat/mb/event.go

 	}
 	if event.Took > 0 {
-		info["rtt"] = event.Took / time.Microsecond
+		e.Put("event.duration", event.Took/time.Microsecond)


This is described as being in nanoseconds in ECS.

@spinscale What will happen if we send nano seconds data to ES versions prior 7?

Good point, will convert it.

Also not related to nano second support in ES as it's not a timestamp.

@andrewkroh As this is a breaking change, I wonder if we want to keep the old field around?

Other option would be to introduce the new field in nano seconds already in 6.6, deprecate the other one and then remove it in 7.0?

The plan you describe sounds good to me. Have both in 6.6 and remove rtt in 7.0.

ruflin · 2018-11-22T14:06:14Z

dev-tools/ecs-migration.yml

+  alias: true
+  copy_to: false
+
+# TODO: This becomes port and domain in servce?


@webmat Feedback welcome on this one.

Hmmm, service doesn't have domain, ip or port.

Is the intent to represent the address exposed by a service? E.g.

service.name: sshd service.port: 22 service.ip: 10.10.10.10 service.domain: (hostname on the local network)

It is the host/hostname that the user configured to connect to the service to fetch the information.

In the case of mysql this can also look like:

hosts: ["root:secret@tcp(127.0.0.1:3306)/"]

I'm thinking to call it uri or connection. Other ideas? I'm more reverting now from splitting this one up and just keep it as defined (without credentials).

I was thinking of introducing a field for these ambiguous addresses as address. It's an extremely generic word, it doesn't clash with any of the words used in the RFC that have a somewhat specific meaning like "host", "hostname", or "authority". address also kinda works for situations where e.g. NGINX or HAProxy shove a unix socket in the "IP" field.

I think "address" could also work for DSNs like the mysql example above.

webmat · 2018-11-23T14:40:43Z

dev-tools/ecs-migration.yml

+
+# TODO: Is it really an alias? Or will we change the unit?
+- from: metricset.rrt
+  to: event.duration


Yeah if this one is changing by a factor of 1 000, I think we can't make this an alias. We'll have to manage the deprecation a bit more manually. Example:

6.x

Mark metricset.rtt as deprecated

Keep populating metricset.rtt field with microseconds

Start populating event.duration field with nanoseconds

Convert all visualizations to use new field

Make sure to correctly set Kibana Index Pattern units for new field

7.0

remove metricset.rtt

Agree on the field migration path. For the dashboards we can't start to use it in 6.x as otherwise it would break with older data.

andrewkroh · 2018-11-28T02:46:27Z

CHANGELOG.asciidoc

@@ -22,6 +22,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha1...master[Check the HEAD d
 *Journalbeat*

 *Metricbeat*
+- event.duration is now in nano and not micro seconds anymore.


s/micro seconds/microseconds/

andrewkroh · 2018-11-28T02:48:14Z

metricbeat/mb/event.go

 	}
 	if event.Took > 0 {
-		info["rtt"] = event.Took / time.Microsecond
+		e.Put("event.duration", event.Took/time.Microsecond)


The plan you describe sounds good to me. Have both in 6.6 and remove rtt in 7.0.

ruflin · 2018-12-05T08:05:14Z

Here is the 6.x PR to introduce event.duration and event.dataset: #9393

ruflin · 2018-12-10T14:56:39Z

dev-tools/ecs-migration.yml

+  breaking: true
+
+- from: metricset.host
+  to: service.uri


@ruflin Note for myself: This should be service.address

ruflin · 2018-12-12T10:17:07Z

jenkins, test this

webmat · 2018-12-12T18:12:10Z

jenkins, test this

webmat

I will say someone much more familiar with these tests should probably review this. I don't feel like I know enough to spot any problems or missing changes.

I did still notice things that seem wrong to me:

Please address the question about service.address
Amidst all of the changes to the test files, there are 3 removals that seem suspicious. They can't be explained by field names or values changing. Are these expected?

First point I think is a potential blocker. Second point, I'm not sure if I'm misunderstanding the situation, this may be expected and not a blocker.

metricbeat/mb/event.go

metricbeat/mb/module/example_test.go

metricbeat/module/system/diskio/_meta/data.json

metricbeat/module/system/process/_meta/data.json

webmat

In this case, this LGTM.

But as I said, someone more familiar with these tests should likely have a look as well.

* Add `event.dataset` as `{module}.{metricset}` but keep `metricset.name` around for backward compatiblity. * Add `event.duration` which comes from `metricset.rtt` but is now in nano instead of micro seconds. `event.duration` will be introduce in 6.x already. * Rename `metricset.module` to `event.module` * Rename `metricset.host` to `service.uri` * Removed `metricset.namespace` as this becomes `event.dataset` and should now always be set. Further changes * Updated some of the data.json files as example * Migration file updated * Old field definitions removed * System tests updated add event field enable ES again fix redis test fix check for host field improve changelog and add todo change uri to address update fields one more update

ruflin added in progress Pull request is currently in progress. Metricbeat Metricbeat ecs labels Nov 6, 2018

ruflin commented Nov 6, 2018

View reviewed changes

andrewkroh reviewed Nov 6, 2018

View reviewed changes

ruflin mentioned this pull request Nov 14, 2018

Transition Beats to ECS #8655

Closed

ruflin force-pushed the migrate-metricset branch 3 times, most recently from 675f8eb to bba0dc1 Compare November 22, 2018 09:41

ruflin commented Nov 22, 2018

View reviewed changes

webmat reviewed Nov 23, 2018

View reviewed changes

ruflin force-pushed the migrate-metricset branch from c283bab to db93785 Compare November 26, 2018 07:17

ruflin added the Team:Integrations Label for the Integrations team label Nov 27, 2018

andrewkroh reviewed Nov 28, 2018

View reviewed changes

alvarolobato assigned ruflin Dec 4, 2018

ruflin force-pushed the migrate-metricset branch 2 times, most recently from c5bbe41 to 382b032 Compare December 5, 2018 07:37

ruflin added review and removed in progress Pull request is currently in progress. labels Dec 5, 2018

ruflin changed the title ~~[WIP] Migration of metricset.* fields to ECS~~ Migration of metricset.* fields to ECS Dec 5, 2018

ruflin force-pushed the migrate-metricset branch 2 times, most recently from 410d653 to 9662a1d Compare December 10, 2018 09:14

ruflin commented Dec 10, 2018

View reviewed changes

webmat mentioned this pull request Dec 10, 2018

Introduce source.address and destination.address. elastic/ecs#247

Merged

ruflin mentioned this pull request Dec 12, 2018

[Auditbeat] Namespace system module to system.audit #9499

Merged

ruflin force-pushed the migrate-metricset branch from a6edeb5 to a7ad64f Compare December 12, 2018 08:09

ruflin force-pushed the migrate-metricset branch from a7ad64f to 53e763b Compare December 12, 2018 15:31

webmat suggested changes Dec 12, 2018

View reviewed changes

webmat approved these changes Dec 12, 2018

View reviewed changes

ruflin force-pushed the migrate-metricset branch 2 times, most recently from 558ba1a to 3459740 Compare December 13, 2018 14:05

ruflin force-pushed the migrate-metricset branch from 3459740 to da2f000 Compare December 13, 2018 16:03

ruflin merged commit 292dd7a into elastic:master Dec 13, 2018

webmat deleted the migrate-metricset branch December 13, 2018 21:07

chrisronline mentioned this pull request Feb 27, 2019

[Add Data] Query for data needs to be updated for Metricbeat elastic/kibana#32154

Open

ChrsMark mentioned this pull request Feb 27, 2020

Add OpenMetrics Metricbeat module #16596

Merged

5 tasks

jsoriano mentioned this pull request Jun 23, 2021

MongoDB: Enable ECS dependency elastic/integrations#1171

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migration of metricset.* fields to ECS #8941

Migration of metricset.* fields to ECS #8941

ruflin commented Nov 6, 2018 •

edited

Loading

ruflin Nov 6, 2018

andrewkroh Nov 6, 2018

ruflin Nov 7, 2018

ruflin Nov 21, 2018

ruflin Nov 22, 2018

andrewkroh Nov 28, 2018

ruflin Nov 22, 2018

webmat Nov 23, 2018

ruflin Nov 29, 2018

ruflin Dec 5, 2018

webmat Dec 5, 2018 •

edited

Loading

webmat Nov 23, 2018 •

edited

Loading

ruflin Dec 5, 2018

andrewkroh Nov 28, 2018

ruflin Dec 5, 2018

andrewkroh Nov 28, 2018

ruflin commented Dec 5, 2018

ruflin Dec 10, 2018

ruflin commented Dec 12, 2018

webmat commented Dec 12, 2018

webmat left a comment

webmat left a comment

Migration of metricset.* fields to ECS #8941

Migration of metricset.* fields to ECS #8941

Conversation

ruflin commented Nov 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

webmat Nov 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Dec 5, 2018

Choose a reason for hiding this comment

ruflin commented Dec 12, 2018

webmat commented Dec 12, 2018

webmat left a comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

ruflin commented Nov 6, 2018 •

edited

Loading

webmat Dec 5, 2018 •

edited

Loading

webmat Nov 23, 2018 •

edited

Loading