Add generic sysctl collector

[discordianfish]

We should implement a sysctl collector to expose arbitrary sysctl values. For that we need to support the following types of sysctl variables:

Numeric values

Number values get exposed as metric value. The user selects them by using the --collector.sysctl.include flag.

single values

Using --collector.sysctl.include=vm.user_reserve_kbytes:
vm.user_reserve_kbytes = 131072 -> node_sysctl_vm_user_reserve_kbytes 131072

multiple values

A sysctl can contain multiple values, for example:

net.ipv4.tcp_rmem = 4096	131072	6291456

Using --collector.sysctl.include=net.ipv4.tcp_rmem the collector will expose:

node_sysctl_net_ipv4_tcp_rmem{index="0"} 4096
node_sysctl_net_ipv4_tcp_rmem{index="1"} 131072
node_sysctl_net_ipv4_tcp_rmem{index="2"} 6291456

If the indexes have defined meaning like in this case, the values can be mapped to multiple metrics by appending the mapping to the --collector.sysctl.include flag:
Using --collector.sysctl.include=net.ipv4.tcp_rmem:min,default,max the collector will expose:

node_sysctl_net_ipv4_tcp_rmem_min 4096
node_sysctl_net_ipv4_tcp_rmem_default 131072
node_sysctl_net_ipv4_tcp_rmem_max 6291456

String Values

String values need to be exposed as info metric. The user selects them by using the --collector.sysctl.include-info flag.

single values

kernel.core_pattern = core -> node_sysctl_info{key="kernel.core_pattern_info", value="core"} 1

multiple values

Given the following sysctl:

kernel.seccomp.actions_avail = kill_process kill_thread trap errno trace log allow

Setting --collector.sysctl.include-info=kernel.seccomp.actions_avail will yield:

node_sysctl_info{key="kernel.seccomp.actions_avail", index="0", value="kill_process"} 1
node_sysctl_info{key="kernel.seccomp.actions_avail", index="1", value="kill_thread"} 1
...

No string sysctl multi values seem to have well defined indexes, so mapping multiple string values to individual metric isn't something we need to support. We might consider it for future proving though.

[binjip978]

@discordianfish by generic you mean we should read a file or somehow rely on procfs?

So if user provided --collector.sysctl.include=vm.user_reserve_kbytes we should open and read /proc/sys/vm/user_reserve_kbytes, or we should somehow take it from procfs?

[discordianfish]

Ah, this already has been implemented in #2425. Closing