[Fleet] Kibana data views missing field format mapping for integrations

[lalit-satapathy]

Elastic packages has format mapping as part of the field.yml. These map into Kibana numerical field formatter and has values like bytes, percentage etc.

In beats, these formatter was applied to the default format of the fields. Here is an example system metricbeat field mapping and corresponding Kibana default field format is set correctly:

In Integrations, the default format mapping is missing as in Kibana above. Here is an example of system integration field mapping and the corresponding Kibana default field format is not set:

It seems that Kibana field format is not applied to all of the system integration field mapping and possibly for all integrations.
CC: @ishleenk17 @ruflin @andresrc

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ruflin]

Lets also make sure these are part of the package spec if not already the case: https://github.com/elastic/package-spec/blob/main/spec/integration/data_stream/fields/fields.spec.yml Personally I always hoped when we introduced meta info per field (elastic/elasticsearch#33267) eventually Kibana data views would read from this to decide what formatter should be used. Unfortunately AFAIK this never happened.

[lalit-satapathy]

Hi @kpollich,

Any conclusion on how this issue will be addressed?

[ruflin]

Also pinging @drewdaemon on this one as we talked about formatters recently and data views. Ideally, this info would be part of the mapping and data views would just read it out like everything else. Then Fleet would not have to do any special actions on it.

[philippkahr]

I wish we would go at it differently. We know what fields are ECS and we could have a list of that inside Kibana? So whenever someone creates a data view, it would autodiscover that transaction.duration.us is part of our APM defined data and will automatically apply the duration microseconds formatter.