[Metricbeat] - Support reporting of kernel audit subsystem statistics

[kholia]

The kernel audit subsystem metrics are available using the following command,

$ sudo auditctl -s
enabled 1
failure 1
pid 806
rate_limit 0
backlog_limit 16384
lost 0
backlog 7
backlog_wait_time 15000
loginuid_immutable 0 unlocked

It would be great to track to track these metrics over time.

• Doing so allows us to tune our auditd / auditbeat configuration properly for the observed systems.

• An untuned auditd / auditbeat deployment can lead to performance and latency problems. Tracking audit subsystem metrics would be helpful in determining when problems (i.e. lost audit events, latency increases due to consequent backpressure) occur.

It seems that this information is not simply available from file(s) in the /proc directory.

$ sudo strace auditctl -s
...

socket(AF_NETLINK, SOCK_RAW, NETLINK_AUDIT) = 3

sendto(3, {len=16, type=0x3e8 /* NLMSG_??? */, flags=NLM_F_REQUEST|NLM_F_ACK, seq=1, pid=0}, 16, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 16

recvfrom(3, {{len=36, type=NLMSG_ERROR, flags=NLM_F_CAPPED, seq=1, pid=2795}, {error=0, msg={len=16, type=AUDIT_GET, flags=NLM_F_REQUEST|NLM_F_ACK, seq=1, pid=0}}}, 8988, MSG_PEEK|MSG_DONTWAIT, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [12]) = 36

recvfrom(3, {{len=36, type=NLMSG_ERROR, flags=NLM_F_CAPPED, seq=1, pid=2795}, {error=0, msg={len=16, type=AUDIT_GET, flags=NLM_F_REQUEST|NLM_F_ACK, seq=1, pid=0}}}, 8988, MSG_DONTWAIT, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [12]) = 36

select(4, [3], NULL, NULL, {tv_sec=0, tv_usec=100000}) = 1 (in [3], left {tv_sec=0, tv_usec=99998})
recvfrom(3, {{len=56, type=AUDIT_GET, flags=0, seq=1, pid=0}, "\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x26\x03\x00\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00\x00\x00\x58\x00\x00\x00"...}, 8988, MSG_DONTWAIT, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [12]) = 56

See #7157 ([Auditbeat] Avoid having Linux wait on clearing a backlog) for the motivation behind this feature request.

Thanks!

[ruflin]

@andrewkroh I wonder if this would make more sense to also be reported by auditbeat directly?

[andrewkroh]

The lost metric is now collected by Auditbeat and reported through the Beats internal metric subsystem. The value is reported as auditd.kernel_lost. See #7179.

So it will show up in logs, be reported in the monitoring data that goes to ES, and be available over HTTP if http.host is configured.

[kholia]

Great!

Would it also be possible to report the backlog value? It could be helpful in tweaking the Auditbeat configuration for an observed system.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.