Add format to Kibana index pattern

[simitt]

According to elastic/kibana#64949 (comment) following formats should be added:

Field: duration_sla_pct
Formatter: Percentage
Numeral js format pattern: 0,0.[]%

Field: transaction_duration_us
Formatter: Duration
Input format: Microseconds
Output format: Milliseconds
Decimal places: 2

All of the changes need to be made in APM Server fields.yml files, and then be properly processed from libbeat fields creator. Beats supports the Format field and the InputFormat, OutputFormat and OutputPrecision which seem to cover the requirements for these changes.

@elastic/apm-ui there is no field duration_sla_pct in the index pattern/index template in APM Server. Is this some calculated field from the UI?

cc @thomasneirynck

[sorenlouv]

there is no field duration_sla_pct in the index pattern/index template in APM Server. Is this some calculated field from the UI?

No, I talked to @thomasneirynck about that field but I have no idea where it's coming from. I don't think we should rely on it being available to end users.

[thomasneirynck]

thx @simitt for raising. This will help the display of the legend, tooltip-values, and text-labels of the default Observability layers in Maps.

wrt duration_sla_pct, I am not sure where this comes from either. We were able to use it in Maps with the sample data provided to us. Maybe @alexfrancoeur @drewpost @cyrille-leclerc can provide additional insights here as well, where this field exactly comes from.

cc @nreese

[cyrille-leclerc]

wrt duration_sla_pct, I am not sure where this comes from either. We were able to use it in Maps with the sample data provided to us. Maybe @alexfrancoeur @drewpost @cyrille-leclerc can provide additional insights here as well, where this field exactly comes from.

Sorry, I can't help on this

[alexfrancoeur]

@thomasneirynck duration_sla_pct and duration_sla are listed in our demo environment. It doesn't look like they're scripted fields in Kibana, so if they aren't part of the APM servers fields, it must be some post processing. @EthanStrider or @gingerwizard might be able to provide more context here.

It feels like this conversation may be going in a very different direction, but while we have this group, can we get some confirmation on the fields in use? We just merged a feature that utilizes out of the box APM fields and got the 👌 from @drewpost and @cyrille-leclerc, so I thought we were OK here.

There's more detail in this meta issue (elastic/kibana#64445), but I've listed the fields we're using below. It sounds like duration_sla_pct should no longer be an option. Is there anything else that stands out?

APM - Real User Monitoring - Performance & Traffic

Choose performance metric:

• Transaction duration: transaction.duration.us
• SLA percentage: duration_sla_pct

Choose traffic metric:

• Total count: Count of transaction.id
• Unique count: Unique count of transaction.id

APM - Real User Monitoring - Performance

Choose performance metric:

• Transaction duration: transaction.duration.us
• SLA percentage: duration_sla_pct

APM - Real User Monitoring - Traffic

Choose traffic metric:

• Total count: Count of transaction.id
• Unique count: Unique count of transaction.id

[sorenlouv]

transaction.duration.us and transaction.id look good. Only duration_sla_pct stands out.

[gingerwizard]

The field duration_sla_pct was computed at index time for a demo dataset. It is not a field generated by beats or something we'd want to add a APM's fields.yaml.

[alexfrancoeur]

Thanks for confirming @gingerwizard @sqren.

[simitt]

I created a PR for the format changes for transaction.duration.us but I think this change is controversial and it might not be a good idea to apply it. Please see #3771 (comment).

[thomasneirynck]

thx @simitt for the update and linking to the older threads.

Yes, let's see where this goes. I don't know enough about the APM-space to know if this will negatively affect Discover-users of APM-data. The Maps-app right now only formats using index-pattern formatters, so +1 on following general recommendation here. It's just that for labeling our clusters or boundaries, those micro-second labels look really large ;)

[sorenlouv]

I'll go out on a tangent and write my observation of generic Kibana apps vs Solutions:

It's just that for labeling our clusters or boundaries, those micro-second labels look really large

Isn't this where the solution should make an opinionated decision and use the format that makes sense in the given context?
In APM app we sometimes format duration as microseconds, millieseconds, seconds, minutes - we even format it as days some places if that makes the most sense.
In our case, it does not make sense to allow the user to configure this: it's context specific. All the user should be able to configure is the locale (whether to use , or . as separator).

[thomasneirynck]

it's context specific.

agreed on that. and to be clear: I'm not suggesting this needs to change for any of the existing display in APM. I just wanted to highlight that for Map-labels, it helps if we can provide some context to what those values are (e.g. "ms" ...). It's a technical limitation right now that Maps needs to use field-formatters.

This need for special treatment in Solutions is shared btw. E.g. the SIEM-app plugs in their own custom formatters when they are using the MapsEmbeddable to show their custom tooltips on the map.

[alexfrancoeur]

Just to confirm while we're working through the field names. Are these the default geo_ip enriched geo fields for RUM?

client.geo.continent_name
client.geo.country_iso_code
client.geo.location

[sorenlouv]

@alexfrancoeur Yes, that looks correct. You can find all the fields here: https://github.com/elastic/apm-server/blob/d8440540e2d2d6a2e9db74627d92fb3c7587e631/docs/fields.asciidoc

client.geo.city_name
City name.
type: keyword
example: Montreal

client.geo.continent_name
Name of the continent.
type: keyword
example: North America

client.geo.country_iso_code
Country ISO code.
type: keyword
example: CA

client.geo.country_name
Country name.
type: keyword
example: Canada

client.geo.location
Longitude and latitude.
type: geo_point
example: { "lon": -73.614830, "lat": 45.505918 }

client.geo.name
User-defined description of a location, at the level of granularity they care about.Could be the name of their data centers, the floor number, if this describes a local physical entity, city names.
Not typically used in automated geolocation.
type: keyword

client.geo.region_iso_code
Region ISO code.
type: keyword
example: CA-QC

client.geo.region_name
Region name.
type: keyword

[simitt]

@thomasneirynck if you don't object, I'd like to close this issue for above discussed reasons (avoiding confusion by providing a different formater than what the field name actually suggests).

[thomasneirynck]

+1, thx @simitt