Add meta information to fields where this applied

[webmat]

Elasticsearch recently added the possibility to store meta field information.

We should add support in ECS tooling to start specifying this information, for fields where this applies. The initial proposal is to start defining the following, but more may come:

• unit: s (seconds)
• unit: B (bytes)
• format: pct

Tasks:

• Add support in the ECS generator
• Populate meta/unit for all fields where it makes sense

I don't think ECS has a field that tracks percentages at this time. But if it does, we should populate it as well.

[ebeahan]

Related Elasticsearch PR working to document standard metadata entries: elastic/elasticsearch#61941

[ebeahan]

Adding some additional details as metadata entries become more formalized.

Valid values for metric_type:

• gauge
• counter

Valid values for unit:

• percent
• bytes
• a time unit
  • d
  • h
  • m
  • s
  • ms
  • micros
  • nanos

[ebeahan]

Related package spec discussion: elastic/package-spec#42

[webmat]

Implementation underway here elastic/integrations#402

[ebeahan]

Need to clarify if it's meta or _meta at the field level. Question posed in another discussion.

[ebeahan]

Confirmed it's meta at a field level - updating title and description here.

[jsoriano]

This issue has appeared when discussing about the inclusion of dimension fields, here elastic/package-spec#247, but I think that dimensions are a different case. A field can be a dimension in some integration, but not in others, but it should be always used with the same type.

Confirmed it's meta at a field level - updating title and description here.

Regarding this, not sure if this influences the implementation here in ECS, but take into account that the metric type is now a mapping setting, see elastic/elasticsearch#74014

The unit should still be stored in meta.

[ebeahan]

Thanks for the insights, @jsoriano. I've wanted to revisit this topic and align ECS (where it makes sense) with recent developments around metrics/TSDB.