Metricbeat: Add network stats for containerized processes

[andrewkroh]

Add networks stats to the system-process output.

The sample output below was collected from Amazon Linux 2016.03 running Docker 1.11.2.

# This is a containerized process.
[root@ip-10-0-50-226 ec2-user]# cat /proc/24138/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth0:     438       5    0    0    0     0          0         0      648       8    0    0    0     0       0          0

# This is a regular process. The metrics are same as the overall system network stats.
[root@ip-10-0-50-226 ec2-user]# cat /proc/2627/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
vethf345468:     648       8    0    0    0     0          0         0      438       5    0    0    0     0       0          0
    lo: 1662073845 1565671    0    0    0     0          0         0 1662073845 1565671    0    0    0     0       0          0
docker0:    2568      38    0    0    0     0          0         0      438       5    0    0    0     0       0          0
  eth0: 771012855  945219    0    0    0     0          0         0 557250737  664840    0    0    0     0       0          0

# These are the overall system metrics.
[root@ip-10-0-50-226 ec2-user]# cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
vethf345468:     648       8    0    0    0     0          0         0      438       5    0    0    0     0       0          0
    lo: 1662073845 1565671    0    0    0     0          0         0 1662073845 1565671    0    0    0     0       0          0
docker0:    2568      38    0    0    0     0          0         0      438       5    0    0    0     0       0          0
  eth0: 771014279  945239    0    0    0     0          0         0 557253481  664852    0    0    0     0       0          0

[andrewkroh]

I have a working prototype of this feature in my branch at https://github.com/andrewkroh/beats/tree/feature/mb/process-network-stats.

The system process metricset includes the network statistics aggregated across each of the interfaces. I did this to avoid having an array of individual interface metrics. You can filter out interfaces if desired (for example, to avoid including metrics from lo0).

Network interface metrics are only included when the process is in a different network namespace than Metricbeat. This is to avoid sending network metrics when those metrics are actually the same as the overall system network metrics. This is accomplished by comparing the namespace information in /proc/self/ns/net to /proc/[pid]/ns/net. If the inode's are different then they are in different namespaces.

Currently I am only enabling this feature when cgroups: true is configured. This feature is not strictly related to cgroups but you would probably enable them together. I'm on the fence about how to enable this?

[ruflin]

It seems like the network stats are not too many additional metrics per event. That means if available I would ship them by default also when cgroup is diabled even though they are kind of linked together. If we don't want to enable it by default (if available) we should add one more config option. In case we flag this addition as experimental, a config option is required.

[monicasarbu]

++ to add another config option (at least for now).

[tangyong]

whether having any release plan for the feature? thanks!

[ruflin]

@andrewkroh Do you remember if we ever moved forward on this one?

[andrewkroh]

This feature was never added. Nearly all the code necessary to add this has been committed upstream so I think it would be relatively simple to add it now.

I still have my first attempt save at andrewkroh@7dd6ba8.

[jlind23]

Backlog grooming: Closing it for now until further activity, can still be reopened if needed.