systemd input not showing in output

[trstringer]

I have the following configuration:

[SERVICE]                              
    Flush 1                            
    Log_Level debug                    

[INPUT]                                
    Name systemd                       
    Tag host.*                         
    Systemd_Filter _SYSTEMD_UNIT=bashdaemon.service                            

[OUTPUT]                               
    Name stdout                        
    Match *   

For some reason, Fluent Bit isn't picking up the journal entries for the bashdaemon.service unit. This screenshot below (direct link) shows this:

• Top terminal: fluent-bit running the configuration
• Bottom left: cat'ing out the configuration (just for verification purposes)
• Bottom right: journalctl output tailing with a filter on the bashdaemon.service unit

Any thoughts on why Fluent Bit may not be ingesting these systemd unit journal entries? Or how to troubleshoot this?

Thanks in advance!

[trstringer]

For what it's worth, this is reproducible without using a custom systemd unit:

$ ./fluent-bit -i systemd -o stdout -p "_SYSTEMD_UNIT=sshd.service"

Empty output when sshd events are entered into journald.

[edsiper]

the following command works properly:

$ fluent-bit -i systemd -p "_SYSTEMD_UNIT=sshd.service" -o stdout -f 1

note that -p needs to be right after the plugin that you want to modify the property. Adding -p after -o stdout means "set that property to stdout".

[trstringer]

@edsiper ah ok. Unfortunately when I use your exact command, I still get nothing. Looks like the systemd input plugin is not working altogether. Debug output is showing me nothing relevant. What's the way to troubleshoot this? I must be missing something.

Or I should just use the rpm that you package up 😄

[edsiper]

can you try adding a path to the systemd journal files ?, e.g:

$ fluent-bit -i systemd -p path=/var/log/journal -p "_SYSTEMD_UNIT=sshd.service" -o stdout -f 1

[trstringer]

Unfortunately that didn't work either (my path was /run/log/journal).

[edsiper]

what about if you append -vv at the end of the command line, do you see any extra message ?

[edsiper]

@trstringer please give me the details of that Linux distro (specific version) to try to replicate the problem

[trstringer]

@edsiper sorry for the delay. Regarding -vv, doesn't seem like anything to go off of:

Fluent-Bit v0.13.0                     
Copyright (C) Treasure Data            

[2018/02/01 21:02:54] [ info] [engine] started                                 
[2018/02/01 21:02:54] [debug] [router] default match rule systemd.0:stdout.0 

Regarding the Linux machine itself:

• CentOS Linux release 7.4.1708 (Core)
• Linux fluentbitvm2 3.10.0-693.17.1.el7.x86_64 Fix include paths #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[s-kostyaev]

Same here.

~ master ±10 ♦ docker run -v /var/log/journal:/var/log/journal:ro -v /etc/machine-id:/etc/machine-id:ro -it fluent/fluent-bit /fluent-bit/bin/fluent-bit -i systemd -p path=/var/log/journal -p tag='host.*' -o stdout -vv
Fluent-Bit v0.12.12
Copyright (C) Treasure Data

[2018/02/06 03:17:51] [ info] [engine] started
[2018/02/06 03:17:51] [debug] [router] default match rule systemd.0:stdout.0

Host OS ubuntu artful:

~ master ±10 ♦ uname -a
Linux nskdev2 4.13.0-32-generic #35-Ubuntu SMP Thu Jan 25 09:13:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
~ master ±10 ♦ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=17.10
DISTRIB_CODENAME=artful
DISTRIB_DESCRIPTION="Ubuntu 17.10"
~ master ±10 ♦ docker -v
Docker version 18.01.0-ce, build 03596f5
~ master ±10 ♦ systemd --version
systemd 234
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN default-hierarchy=hybrid

In another console:

~ master ±10 ♦ logger "some test message"
~ master ±10 ♦ logger "some test message"

I can see this messages with journalctl:

~ master ±10 ♦ journalctl -f
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
-- Logs begin at Mon 2017-12-18 13:04:29 +07. --
фев 06 10:08:26 nskdev2 feofan[21866]: some test message
фев 06 10:13:30 nskdev2 feofan[523]: some test message
фев 06 10:13:32 nskdev2 feofan[580]: some test message
фев 06 10:13:33 nskdev2 feofan[643]: some test message
фев 06 10:16:23 nskdev2 feofan[7177]: some test message
фев 06 10:16:24 nskdev2 feofan[7229]: some test message
фев 06 10:17:15 nskdev2 feofan[9418]: some test message
фев 06 10:17:26 nskdev2 feofan[9862]: some test message
фев 06 10:17:54 nskdev2 feofan[11087]: some test message
фев 06 10:17:55 nskdev2 feofan[11278]: some test message
фев 06 10:21:05 nskdev2 feofan[18115]: some test message

Also with strace I can see memmap & memunmap of updated journal file.

[s-kostyaev]

On this machine:

# uname -a
Linux k8s-dev-node-1 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
# docker -v
Docker version 17.03.1-ce, build c6d412e
# systemd --version
systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

all works fine

[edsiper]

@trstringer @s-kostyaev

there is a new fix associated to systemd, would you please rebuild GIT master and test if the problem you describe is fixed ?

ref: #500

[s-kostyaev]

@edsiper After rebuilding docker image from fluent-bit git master problem is not solved:

Fluent-Bit v0.13.0
Copyright (C) Treasure Data

[2018/02/07 03:33:35] [ info] [engine] started
[2018/02/07 03:33:35] [debug] [router] default match rule systemd.0:stdout.0

[edsiper]

@s-kostyaev did you check that journalctl works inside the docker image ?