Consider combining "<type>" and "state_<type>" kubernetes metricsets into a single metricset/event.

[dicarlo2]

Looking at #6158 it seems like metricbeat is heading in that direction. It'd be convenient for certain visualizations (e.g. tables) to have all of the metrics in a single document from both the kubelet and kube-state-metrics for nodes, pods and containers. @exekias

[exekias]

I agree with this idea, the main problem is we face some technical challenges to do so:

For instance, metricsets are configured against a single host, in this case of kubernetes container and state_container, they talk with different services (kubelet + kube-state-metrics), so somehow we would need to add a second host setting for kube-state-metrics.

Also, kube-sate-metrics may not be available, and we can graceful fallback to ignore that, but it may break user expectations. While with the current approach you would get a common error as with any other broken host settings.

I'm wondering what's the use case you are thinking of to require both together?

[dicarlo2]

An example use case I'm thinking of, and perhaps it's naive because I'm a relatively new to using elasticsearch/kibana, is to have a a table that shows the latest state of all containers in terms of both the kubelet metrics (e.g. cpu/memory utilization) as well as the kube-state-metrics (e.g. container ready status) - one row per container.

On that note, another metric I'd like to see is an aggregation of the container ready status in the pod document similar to kubectl get pods. Again, maybe that's possible with some sort of aggregation. Does that make sense? If there's a way to achieve that already with an elasticsearch query and the existing data, I'd love to know how :)

[dicarlo2]

Also, I'm curious about the order of events with the current solution, it seems like there could be a race condition there. It doesn't really matter for the current metrics since we can probably assume limits are not changing frequently, but for example, pulling the current ready status into the main container document seems like it could be off by one period depending on if the kube-state-metrics fetch occurs before or after the kubelet fetch.

[exekias]

we finally went down this other path: #7470, I don't think merging this metricsets is worth it as of today, as they are now sharing all the metadata, meaning the resulting metrics can be searched in the same way.

Merging them has some added costs on how we query for metrics.