Allow for modifying json field names with dots in them in beats

[dl1ely]

Following the suggestion in https://discuss.elastic.co/t/allow-for-renaming-of-json-fields-in-http-module-output/112461 i open this issue to discuss the challenges regarding json field names with dots in them when these get collected/created by beats, especially metricbeat.

The problem is also described in https://discuss.elastic.co/t/v6-0-0-beta2-http-module-mapping-trouble/99703?source_topic_id=112461 and was already addressed in httpbeat, the predecessor of the metricbeat http module (https://github.com/christiangalsterer/httpbeat/blob/master/config/config.go#L34). Unfortunately, the http module does not allow a replacement of dots in json keys collected from the source.

The issue also surfaced in Spring Boot Actuator spring-projects/spring-boot#10449 but it affects other producers of metrics data too, and will not be fixed in Spring Boot 1.5

[christiangalsterer]

@ruflin: If you have an idea/preference/suggestion how to address this I can have a look into it.

[ruflin]

I think this is actually problem we have to solve in different places but probably have a method internally to do it. In general we should have a method replacing dot separated keys with _ as we already do for example for docker labels.

Here the different fixes we should do:

• Jolokia: We should do it here by default as part of the metricset as it happens too often
• http json metricset: Have a config option there
• JSON processor: We should allow to do this as an option there too, which probably internally can be used by the http/ json metricset.

@christiangalsterer Would be great to have a PR for the http/json metricset, I will have a look at the jolokia one.

[ruflin]

BTW here is the discussion from the Jolokia side: https://discuss.elastic.co/t/include-fields-problem/109809

[ruflin]

@christiangalsterer You can use the dedot function introduced here: #5957

[christiangalsterer]

@ruflin: Thanks for the info. Will have a look into the http/json metricset.

[ruflin]

@christiangalsterer Awesome. Ping me if I can be of any help.

[walktall]

Just like docker labels, add_kubernetes_metadata processors also need to handle k8s labels and annotations.

This method should also apply common.DeDot introduced in #5957 for keys of labelMap and annotationsMap.

[christiangalsterer]

PR #5970 available for http/json module.

[ruflin]

Based on a discussion started about dots here (micrometer-metrics/micrometer#154 (comment)) I'm now thinking we probably need an additional conversion options to keep the dots but add .value or .count to the keys which conflict.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.