[Auditbeat] test config command with system socket dataset fails if another auditbeat is running

[ansell]

Please post all questions and issues on https://discuss.elastic.co/c/beats
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.

For security vulnerabilities please only send reports to security@elastic.co.
See https://www.elastic.co/community/security for more information.

Please include configurations and logs if available.

For confirmed bugs, please report:

• Version: auditbeat-7.5.2
• Operating System: Ubuntu Linux (tested with 14.04/16.04/18.04)
• Discuss Forum URL: https://discuss.elastic.co/t/auditbeat-test-config-fails-when-auditbeat-already-running/216312 (No response to the community forum question and no indication that it reached a relevant audience, so opening an issue here as I have a minimal testcase to replicate it myself on different servers and do not believe it is related to my particular setup)
• Steps to Reproduce:
  Run auditbeat using the following configuration, verifying it is picking up system socket events. Then, while auditbeat is running, attempt to test the same configuration using /path/to/auditbeat/bin/auditbeat test config -c auditbeat-config.yml

auditbeat.modules:

- module: system
  datasets:
    - socket

name: auditbeat-bug-minimal

tags: ["auditbeat-bug-minimal"]

logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/auditbeat
  name: auditbeat
  keepfiles: 7
  rotateeverybytes: 10485760 # = 10MB

output.file:
  enabled: true
  codec.json:
    pretty: true
  path: "/tmp/auditbeat"

There are no error messages when running auditbeat using that configuration. However, attempting to test the configuration on the same server gives the following error, indicating that when the system module with the socket dataset is enabled, the test configuration attempts to remove the kernel probes for the existing running instance before the configuration test is able to complete:

# /usr/share/auditbeat/bin/auditbeat test config -c auditbeat-minimal.yml 
Exiting: 1 error: 1 error: system/socket dataset setup failed: unable to delete existing KProbes. Is Auditbeat already running?: 23 errors: unable to remove kprobe 'p:auditbeat/sys_execve_call SyS_execve path=+0(%di):u64 arg2=+8(%di):u64 arg3=+16(%di):u64 arg4=+24(%di):u64 arg5=+32(%di):u64 arg6=+40(%di):u64 arg7=+48(%di):u64 arg8=+56(%di):u64 arg9=+64(%di):u64 arg10=+72(%di):u64 arg11=+80(%di):u64 arg12=+88(%di):u64 arg13=+96(%di):u64 arg14=+104(%di):u64 arg15=+112(%di):u64 arg16=+120(%di):u64 argptrs=+0(%si):u64 arg18=+8(%si):u64 arg19=+16(%si):u64 arg20=+24(%si):u64 arg21=+32(%si):u64 arg22=+40(%si):u64 param0=+0(+0(%si)):u64 arg24=+8(+0(%si)):u64 arg25=+16(+0(%si)):u64 arg26=+24(+0(%si)):u64 arg27=+32(+0(%si)):u64 arg28=+40(+0(%si)):u64 arg29=+48(+0(%si)):u64 arg30=+56(+0(%si)):u64 arg31=+64(+0(%si)):u64 arg32=+72(+0(%si)):u64 arg33=+80(+0(%si)):u64 arg34=+88(+0(%si)):u64 arg35=+96(+0(%si)):u64 arg36=+104(+0(%si)):u64 arg37=+112(+0(%si)):u64 arg38=+120(+0(%si)):u64 param1=+0(+8(%si)):u64 arg40=+8(+8(%si)):u64 arg41=+16(+8(%si)):u64 arg42=+24(+8(%si)):u64 arg43=+32(+8(%si)):u64 arg44=+40(+8(%si)):u64 arg45=+48(+8(%si)):u64 arg46=+56(+8(%si)):u64 arg47=+64(+8(%si)):u64 arg48=+72(+8(%si)):u64 arg49=+80(+8(%si)):u64 arg50=+88(+8(%si)):u64 arg51=+96(+8(%si)):u64 arg52=+104(+8(%si)):u64 arg53=+112(+8(%si)):u64 arg54=+120(+8(%si)):u64 param2=+0(+16(%si)):u64 arg56=+8(+16(%si)):u64 arg57=+16(+16(%si)):u64 arg58=+24(+16(%si)):u64 arg59=+32(+16(%si)):u64 arg60=+40(+16(%si)):u64 arg61=+48(+16(%si)):u64 arg62=+56(+16(%si)):u64 arg63=+64(+16(%si)):u64 arg64=+72(+16(%si)):u64 arg65=+80(+16(%si)):u64 arg66=+88(+16(%si)):u64 arg67=+96(+16(%si)):u64 arg68=+104(+16(%si)):u64 arg69=+112(+16(%si)):u64 arg70=+120(+16(%si)):u64 param3=+0(+24(%si)):u64 arg72=+8(+24(%si)):u64 arg73=+16(+24(%si)):u64 arg74=+24(+24(%si)):u64 arg75=+32(+24(%si)):u64 arg76=+40(+24(%si)):u64 arg77=+48(+24(%si)):u64 arg78=+56(+24(%si)):u64 arg79=+64(+24(%si)):u64 arg80=+72(+24(%si)):u64 arg81=+80(+24(%si)):u64 arg82=+88(+24(%si)):u64 arg83=+96(+24(%si)):u64 arg84=+104(+24(%si)):u64 arg85=+112(+24(%si)):u64 arg86=+120(+24(%si)):u64 param4=+0(+32(%si)):u64 arg88=+8(+32(%si)):u64 arg89=+16(+32(%si)):u64 arg90=+24(+32(%si)):u64 arg91=+32(+32(%si)):u64 arg92=+40(+32(%si)):u64 arg93=+48(+32(%si)):u64 arg94=+56(+32(%si)):u64 arg95=+64(+32(%si)):u64 arg96=+72(+32(%si)):u64 arg97=+80(+32(%si)):u64 arg98=+88(+32(%si)):u64 arg99=+96(+32(%si)):u64 arg100=+104(+32(%si)):u64 arg101=+112(+32(%si)):u64 arg102=+120(+32(%si)):u64': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'r:auditbeat/sys_execve_ret SyS_execve retval=%ax:s32': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/do_exit do_exit ': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/commit_creds commit_creds uid=+4(%di):u32 gid=+8(%di):u32 euid=+20(%di):u32 egid=+24(%di):u32': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/sock_init_data sock_init_data socket=%di sock=%si': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/inet_create inet_create proto=%dx:s32': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/inet_release inet_release sock=+32(%di)': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/tcp4_connect_in tcp_v4_connect sock=%di laddr=+4(%di):u32 lport=+728(%di):u16 af=+0(%si):u16 addr=+4(%si):u32 port=+2(%si):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'r:auditbeat/tcp4_connect_out tcp_v4_connect retval=%ax:s32': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/ip_local_out_call ip_local_out sock=%si size=+128(%dx):u32 af=+16(%si):u16 laddr=+4(%si):u32 lport=+728(%si):u16 raddr=+0(%si):u32 rport=+12(%si):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/tcp_v4_do_rcv_call tcp_v4_do_rcv sock=%di size=+128(%si):u32 laddr=+4(%di):u32 lport=+728(%di):u16 raddr=+0(%di):u32 rport=+12(%di):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/udp_sendmsg_in udp_sendmsg sock=%di size=%dx laddr=+4(%di):u32 lport=+728(%di):u16 raddr=+4(+0(%si)):u32 rport=+2(+0(%si)):u16 altraddr=+0(%di):u32 altrport=+12(%di):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/udp_queue_rcv_skb udp_queue_rcv_skb sock=%di size=+128(%si):u32 laddr=+4(%di):u32 lport=+728(%di):u16 iphdr=+196(%si):u16 udphdr=+194(%si):u16 base=+208(%si) packet=+0(+208(%si)):u64 arg9=+8(+208(%si)):u64 arg10=+16(+208(%si)):u64 arg11=+24(+208(%si)):u64 arg12=+32(+208(%si)):u64 arg13=+40(+208(%si)):u64 arg14=+48(+208(%si)):u64 arg15=+56(+208(%si)):u64 arg16=+64(+208(%si)):u64 arg17=+72(+208(%si)):u64 arg18=+80(+208(%si)):u64 arg19=+88(+208(%si)):u64 arg20=+96(+208(%si)):u64 arg21=+104(+208(%si)):u64 arg22=+112(+208(%si)):u64 arg23=+120(+208(%si)):u64 arg24=+128(+208(%si)):u64 arg25=+136(+208(%si)):u64 arg26=+144(+208(%si)):u64 arg27=+152(+208(%si)):u64 arg28=+160(+208(%si)):u64 arg29=+168(+208(%si)):u64 arg30=+176(+208(%si)):u64 arg31=+184(+208(%si)):u64 arg32=+192(+208(%si)):u64 arg33=+200(+208(%si)):u64 arg34=+208(+208(%si)):u64 arg35=+216(+208(%si)):u64 arg36=+224(+208(%si)):u64 arg37=+232(+208(%si)):u64 arg38=+240(+208(%si)):u64 arg39=+248(+208(%si)):u64': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/clock_sync_probe SyS_newuname magic=+0(%di):u64 timestamp=+8(%di):u64': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/inet6_create inet6_create proto=%dx:s32': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/inet6_csk_xmit_call inet6_csk_xmit sock=%di size=+128(%si):u32 lport=+728(%di):u16 rport=+12(%di):u16 laddr6a=+72(%di):u64 laddr6b=+80(%di):u64 raddr6a=+56(%di):u64 raddr6b=+64(%di):u64': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/tcp_v6_do_rcv_call tcp_v6_do_rcv sock=%di size=+128(%si):u32 lport=+728(%di):u16 rport=+12(%di):u16 laddr6a=+72(%di):u64 laddr6b=+80(%di):u64 raddr6a=+56(%di):u64 raddr6b=+64(%di):u64': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/tcp6_connect_in tcp_v6_connect sock=%di laddra=+72(%di):u64 laddrb=+80(%di):u64 lport=+728(%di):u16 af=+0(%si):u16 addra=+8(%si):u64 addrb=+16(%si):u64 port=+2(%si):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'r:auditbeat/tcp6_connect_out tcp_v6_connect retval=%ax:s32': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/udpv6_sendmsg_in udpv6_sendmsg sock=%di size=%dx laddra=+72(%di):u64 laddrb=+80(%di):u64 lport=+728(%di):u16 raddra=+8(+0(%si)):u64 raddrb=+16(+0(%si)):u64 rport=+2(+0(%si)):u16 altraddra=+56(%di):u64 altraddrb=+64(%di):u64 altrport=+12(%di):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/udpv6_queue_rcv_skb udpv6_queue_rcv_skb sock=%di size=+128(%si):u32 laddra=+72(%di):u64 laddrb=+80(%di):u64 lport=+728(%di):u16 iphdr=+196(%si):u16 udphdr=+194(%si):u16 base=+208(%si) packet=+0(+208(%si)):u64 arg10=+8(+208(%si)):u64 arg11=+16(+208(%si)):u64 arg12=+24(+208(%si)):u64 arg13=+32(+208(%si)):u64 arg14=+40(+208(%si)):u64 arg15=+48(+208(%si)):u64 arg16=+56(+208(%si)):u64 arg17=+64(+208(%si)):u64 arg18=+72(+208(%si)):u64 arg19=+80(+208(%si)):u64 arg20=+88(+208(%si)):u64 arg21=+96(+208(%si)):u64 arg22=+104(+208(%si)):u64 arg23=+112(+208(%si)):u64 arg24=+120(+208(%si)):u64 arg25=+128(+208(%si)):u64 arg26=+136(+208(%si)):u64 arg27=+144(+208(%si)):u64 arg28=+152(+208(%si)):u64 arg29=+160(+208(%si)):u64 arg30=+168(+208(%si)):u64 arg31=+176(+208(%si)):u64 arg32=+184(+208(%si)):u64 arg33=+192(+208(%si)):u64 arg34=+200(+208(%si)):u64 arg35=+208(+208(%si)):u64 arg36=+216(+208(%si)):u64 arg37=+224(+208(%si)):u64 arg38=+232(+208(%si)):u64 arg39=+240(+208(%si)):u64 arg40=+248(+208(%si)):u64': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/tcp_sendmsg_in tcp_sendmsg sock=%di size=%dx laddr=+4(%di):u32 lport=+728(%di):u16 raddr=+0(%di):u32 rport=+12(%di):u16 family=+16(%di):u16 laddr6a=+72(%di):u64 laddr6b=+80(%di):u64 raddr6a=+56(%di):u64 raddr6b=+64(%di):u64': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'r:auditbeat/inet_csk_accept_ret inet_csk_accept sock=%ax laddr=+4(%ax):u32 lport=+728(%ax):u16 raddr=+0(%ax):u32 rport=+12(%ax):u16 family=+16(%ax):u16 laddr6a=+72(%ax):u64 laddr6b=+80(%ax):u64 raddr6a=+56(%ax):u64 raddr6b=+64(%ax):u64': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy

This live config testing before swapping in on the same server works with other auditbeat datasets and other beats.

Config testing when the system socket dataset is not enabled works, returning the following message:

# /usr/share/auditbeat/bin/auditbeat test config -c auditbeat-minimal.yml 
Config OK

[ansell]

Is it possible to get some indication of whether this has been triaged?

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.