3.1.3.2 Publishing system and cluster metrics using Netdata

This method of acquiring system and K8S metric values involves the deployment of one Netdata agent at every K8S cluster node. Netdata is an open source software for collecting metrics, displaying them as charts, but also providing them through a REST API. The default Nebulous application deployment scenario installs Netdata agents along with EMS at application clusters. EPAs will periodically contact the REST API server of each Netdata agent and scrape the required metrics. To enable EPAs scrape the Netdata agents, it is required that the application metric model provides the needed configuration. For each raw metric that will have its values using this method, it is necessary to define a sensor of “netdata” type and provide the corresponding configuration (including the scraping period).

In order to define a raw metric that takes its values from Netdata agents, the netdata type must be entered in the Sensor field in Nebulous GUI. This will instruct EPA to use its K8S Netdata collector plugin for retrieving the values. Under the hood the K8S Netdata collector plugin will build a URL of the form http://<NODE_IP_ADDRESS>:<PORT>/<PATH>?<QUERY_PARAMS>&format=ssv and attempt to retrieve the relevent JSON response. Following, it will extract the value(s) of the metric of interest (see next) and publish it/them as the raw metric's value(s) in EPA broker. If needed, it will also aggreate multiple values into a single one. In order to build the URL, the collector plugin will use the provided configuration settings, or the corresponding defaults. If the metric of interest is a K8S metric (its names starts with k8s.) the collector plugin can take into consideration the pod name and namespace. The metric(s) of interest must be provided in the configuration.

The configuration comprises a few settings used to guide the collector plugin, while the remaining are used to build the QUERY_PARAMS list of the URL. The plugin-specific configuration settings, along with their respective defaults, are:

Setting	Type	Default value	Comments
endpoint	String	/api/v2/data	The <PATH> part of the URL. Only the v2 version has been tested.
port	Port	19999	The <PORT> part of the URL. Allowed values: 1..65535.
components	String	component name	In case of K8S metric of interest, specifies which pod(s) to pick. If left empty it will pick all pods in the namespace. If omitted it will use the name of component(s) the raw metric applies.
namespace	String	default	In case of K8S metric of interest, specifies the pod namespace to use. If left empty or omitted it defaults to default namespace.
results-aggregation	Enum	no default	Allowed values: SUM, AVERAGE, COUNT, MIN, MAX, NONE. If omitted or is NONE individual events will be published for each metric value.
intervalPeriod	Positive Integer	60	How often to scrape the prometheus/OpenMetrics endpoint.
intervalUnit	Enum	SECONDS	The time unit of intervalPeriod. Allowed values: SECONDS, MINUTES, HOURS, DAYS
If intervalPeriod is omitted the scraping period is taken from the metric's Output interval and unit fields. If they are not specified either, it is assumed to be 60 seconds.

The settings used to build the Netdata URL, along with their respective defaults, are:

Setting	Type	Default value	Comments
scope_contexts	String	no default	REQUIRED: The metric(s) of interest to extract. Can be a comma-separated list
context	String	no default	Can be used instead of scope_contexts. Check Netdata documentation for details
dimension	String	*	The scope_context dimensions to use
after	Long	-1	.....
time_group	Enum	average	.....
group_by	Enum	label	In case of K8S metric of interest, .....
group_by_label	String	k8s_namespace,k8s_pod_name	In case of K8S metric of interest, .....

The settings in the table above are always added in the URL (either the provided value or the default). Any additional settings (not listed in tables above) will also be included in query parameters list.

For a complete list of the supported query parameters, and their semantics, please consult teh official Netdata documentation. LINK++++++++++

---

Appendix

Sample list of Netdata metrics. It may vary between devices based on the hardware, architecture, OS, but also the installed software.

..........TBD

---

Additional reading

[1] Netdata site