falco logs contaminating /var/log/messages #1673

ostendali · 2021-06-09T17:12:38Z

Describe the bug
falco writing logs in /var/log/messages and contaminating syslogs.
After having changed the configuration in /etc/falco/falco.yaml by disabling syslog_output:

syslog_output:  
  enabled: false

Falco seems to ignore this configuration and continues to write to /var/log/messages

How to reproduce it

add falco repositories to systems curl -s -o /etc/yum.repos.d/falcosecurity.repo https://falco.org/repo/falcosecurity-rpm.repo
install falco i.e. deb/rpm yum install falco (or apt install falco)
check the '/var/log/messages`

Expected behaviour
After disabling syslog_output falco should not write logs into /var/log/messages

Environment

Falco version: 0.28.2
System info:

{
 "machine": "x86_64",
 "nodename": "demo-milan.acsia.io",
 "release": "3.10.0-1160.11.1.el7.x86_64",
 "sysname": "Linux",
 "version": "#1 SMP Fri Dec 18 16:34:56 UTC 2020"
}

Cloud provider or hardware configuration: AWS
OS: CentOS/Debian/Ubuntu/Suse
Kernel:3.10.0-1160.11.1.el7.x86_64
Installation method: RPM/DEB from repo

This creates 2 particular issues:

contaminating /var/log/messages (compliance)
generating big volume of logs in messages (syslog)

The text was updated successfully, but these errors were encountered:

domenico4sec · 2021-06-17T08:45:40Z

I have found the same bug in the last version of Falco 0.28.1 (after the packages 0.28.2 was removed from Falco repos)

leodido · 2021-06-17T13:12:19Z

I used Falco 0.28.1-34+0f24448 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d) on the following system:

{
  "machine": "x86_64",
  "nodename": "REDACTED",
  "release": "5.4.0-1045-aws",
  "sysname": "Linux",
  "version": "#47-Ubuntu SMP Tue Apr 13 07:02:25 UTC 2021"
}

Disabling syslog_output I'm not able to reproduce this behavior. Meaning, I don't see Falco alerts (even if I generated some I can see in the stdout) in /var/log/messages, I only see boot/shutdown Falco outputs in /var/log/messages.

Jun 17 13:10:00 ip-172-31-46-109 falco: Falco version 0.28.1-34+0f24448 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d)
Jun 17 13:10:00 ip-172-31-46-109 falco: Falco initialized with configuration file /etc/falco/falco.yaml
Jun 17 13:10:00 ip-172-31-46-109 falco: Loading rules from file /etc/falco/falco_rules.yaml:
Jun 17 13:10:00 ip-172-31-46-109 falco: Loading rules from file /etc/falco/falco_rules.local.yaml:
Jun 17 13:10:00 ip-172-31-46-109 falco: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.451281] falco: adding new consumer 00000000fea1a8d2
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.451294] falco: initializing ring buffer for CPU 0
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.475657] falco: CPU buffer initialized, size=8388608
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.475660] falco: initializing ring buffer for CPU 1
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.504736] falco: CPU buffer initialized, size=8388608
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.504738] falco: initializing ring buffer for CPU 2
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.538759] falco: CPU buffer initialized, size=8388608
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.538761] falco: initializing ring buffer for CPU 3
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.571165] falco: CPU buffer initialized, size=8388608
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.571168] falco: initializing ring buffer for CPU 4
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.599411] falco: CPU buffer initialized, size=8388608
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.599413] falco: initializing ring buffer for CPU 5
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.622187] falco: CPU buffer initialized, size=8388608
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.622189] falco: initializing ring buffer for CPU 6
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.648849] falco: CPU buffer initialized, size=8388608
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.648852] falco: initializing ring buffer for CPU 7
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.675664] falco: CPU buffer initialized, size=8388608
Jun 17 13:10:01 ip-172-31-46-109 kernel: [1127042.675667] falco: starting capture
Jun 17 13:10:01 ip-172-31-46-109 falco: Starting internal webserver, listening on port 8765
Jun 17 13:10:07 ip-172-31-46-109 falco: SIGINT received, exiting...
Jun 17 13:10:07 ip-172-31-46-109 kernel: [1127048.565977] falco: deallocating consumer 00000000fea1a8d2
Jun 17 13:10:07 ip-172-31-46-109 kernel: [1127048.581148] falco: no more consumers, stopping capture

Are you referring to all the Falco logs or only to the Falco alerts?

domenico4sec · 2021-06-17T14:06:44Z

Hi Leo, thank you for your answer.

Are you referring to all the Falco logs or only to the Falco alerts?

All Falco logs AND Falco alerts are written in var/log/messages

I am going to do other tests now and add as much details as possible in another comment.

domenico4sec · 2021-06-17T15:21:11Z

I did a test from a clean AWS instance and used the installation script

Machine

I used AWS to perform this test:
CentOS 7 (x86_64) - with Updates HVM
https://aws.amazon.com/marketplace/pp/prodview-qkzypm3vjr45g?ref=cns_srchrow

Prepare machine

update and restart

# sudo yum update -y
# sudo reboot now

OS info

# cat /etc/*release*
CentOS Linux release 7.9.2009 (Core)
Derived from Red Hat Enterprise Linux 7.9 (Source)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.9.2009 (Core)
CentOS Linux release 7.9.2009 (Core)
cpe:/o:centos:centos:7

Installation

following https://falco.org/docs/getting-started/third-party/install-tools/

# curl -o install_falco -s https://falco.org/script/install
# sudo bash install_falco

Installation output

* Detecting operating system
* Installing EPEL repository (for DKMS)
warning: /var/tmp/rpm-tmp.EhxjMc: # curl -o install_falco -s https://falco.org/script/installHeader V4 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
* Installing Falco public GPG key
* Installing Falco repository
* Installing kernel headers
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
Warning: RPMDB altered outside of yum.
* Installing Falco
warning: /var/cache/yum/x86_64/7/epel/packages/dkms-2.8.4-1.el7.noarch.rpm: Header V4 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Public key for dkms-2.8.4-1.el7.noarch.rpm is not installed
Importing GPG key 0x352C64E5:
 Userid     : "Fedora EPEL (7) <epel@fedoraproject.org>"
 Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
 Package    : epel-release-7-13.noarch (installed)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

Creating symlink /var/lib/dkms/falco/5c0b863ddade7a45568c0ac97d037422c9efb750/source ->
                 /usr/src/falco-5c0b863ddade7a45568c0ac97d037422c9efb750

DKMS: add completed.

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area...
make -j2 KERNELRELEASE=3.10.0-1160.31.1.el7.x86_64 -C /lib/modules/3.10.0-1160.31.1.el7.x86_64/build M=/var/lib/dkms/falco/5c0b863ddade7a45568c0ac97d037422c9efb750/build....
cleaning build area...

DKMS: build completed.

falco.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.10.0-1160.31.1.el7.x86_64/extra/
Adding any weak-modules

depmod....

DKMS: install completed.
modprobe: FATAL: Module falco_probe not found.

Update Falco configuration

In /etc/falco/falco.yaml disable syslog output

syslog_output:
  enabled: false

Check and start Falco

Check status

# systemctl status falco -l
● falco.service - Falco: Container Native Runtime Security
   Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: https://falco.org/docs/

Jun 17 14:52:15 ip-172-31-36-116.eu-west-3.compute.internal systemd[1]: [/usr/lib/systemd/system/falco.service:19] Unknown lvalue 'ProtectKernelTunables' in section 'Service'
Jun 17 14:52:15 ip-172-31-36-116.eu-west-3.compute.internal systemd[1]: [/usr/lib/systemd/system/falco.service:20] Unknown lvalue 'RestrictRealtime' in section 'Service'
Jun 17 14:52:27 ip-172-31-36-116.eu-west-3.compute.internal systemd[1]: [/usr/lib/systemd/system/falco.service:19] Unknown lvalue 'ProtectKernelTunables' in section 'Service'
Jun 17 14:52:27 ip-172-31-36-116.eu-west-3.compute.internal systemd[1]: [/usr/lib/systemd/system/falco.service:20] Unknown lvalue 'RestrictRealtime' in section 'Service'
Jun 17 14:52:36 ip-172-31-36-116.eu-west-3.compute.internal systemd[1]: [/usr/lib/systemd/system/falco.service:19] Unknown lvalue 'ProtectKernelTunables' in section 'Service'
Jun 17 14:52:36 ip-172-31-36-116.eu-west-3.compute.internal systemd[1]: [/usr/lib/systemd/system/falco.service:20] Unknown lvalue 'RestrictRealtime' in section 'Service'

Start Falco

# systemctl status falco -l
● falco.service - Falco: Container Native Runtime Security
   Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-06-17 14:53:08 UTC; 1min 8s ago
     Docs: https://falco.org/docs/
  Process: 12338 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
 Main PID: 12342 (falco)
   CGroup: /system.slice/falco.service
           └─12342 /usr/bin/falco --pidfile=/var/run/falco.pid

Jun 17 14:53:08 ip-172-31-36-116.eu-west-3.compute.internal falco[12342]: Thu Jun 17 14:53:08 2021: Falco initialized with configuration file /etc/falco/falco.yaml
Jun 17 14:53:08 ip-172-31-36-116.eu-west-3.compute.internal falco[12342]: Thu Jun 17 14:53:08 2021: Loading rules from file /etc/falco/falco_rules.yaml:
Jun 17 14:53:08 ip-172-31-36-116.eu-west-3.compute.internal falco[12342]: Falco initialized with configuration file /etc/falco/falco.yaml
Jun 17 14:53:08 ip-172-31-36-116.eu-west-3.compute.internal falco[12342]: Loading rules from file /etc/falco/falco_rules.yaml:
Jun 17 14:53:08 ip-172-31-36-116.eu-west-3.compute.internal falco[12342]: Loading rules from file /etc/falco/falco_rules.local.yaml:
Jun 17 14:53:08 ip-172-31-36-116.eu-west-3.compute.internal falco[12342]: Thu Jun 17 14:53:08 2021: Loading rules from file /etc/falco/falco_rules.local.yaml:
Jun 17 14:53:09 ip-172-31-36-116.eu-west-3.compute.internal falco[12342]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Jun 17 14:53:09 ip-172-31-36-116.eu-west-3.compute.internal falco[12342]: Thu Jun 17 14:53:09 2021: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Jun 17 14:53:09 ip-172-31-36-116.eu-west-3.compute.internal falco[12342]: Starting internal webserver, listening on port 8765
Jun 17 14:53:09 ip-172-31-36-116.eu-west-3.compute.internal falco[12342]: Thu Jun 17 14:53:09 2021: Starting internal webserver, listening on port 8765

Reproduce error

Generate alert

# touch /etc/domenico

log in /var/log/messages

Jun 17 14:55:23 ip-172-31-36-116 falco: 14:55:23.315542901: Error File below /etc opened for writing (user=root user_loginuid=1000 command=touch /etc/domenico parent=bash pcmdline=bash file=/etc/domenico program=touch gparent=sudo ggparent=bash gggparent=sshd container_id=host image=<NA>)
Jun 17 14:55:23 ip-172-31-36-116 falco: 14:55:23.315542901: Error File below /etc opened for writing (user=root user_loginuid=1000 command=touch /etc/domenico parent=bash pcmdline=bash file=/etc/domenico program=touch gparent=sudo ggparent=bash gggparent=sshd container_id=host image=<NA>)

/cc @leodido

domenico4sec · 2021-06-18T15:23:21Z

It looks like enabling stdout_output (enabled by default) makes falco logs in var/log/messages

We have an old machine with falco 0.20 with stdout_output enabled and falco does not write in messages, so something changed between version 0.21 and 0.28.

leogr · 2021-06-23T11:02:34Z

It looks like enabling stdout_output (enabled by default) makes falco logs in var/log/messages

We have an old machine with falco 0.20 with stdout_output enabled and falco does not write in messages, so something changed between version 0.21 and 0.28.

Since 0.28 we migrated from initd to systemd 👉 #1448
I haven't tried yet with systemd, but my guess is that systemd is copying the stdout of Falco to /var/log/messages.

The behavior should be configurable by the systemd unit configuration (see StandardOutput= in https://manpages.debian.org/wheezy/systemd/systemd.exec.5.en.html)

Also, I can confirm the problem is not present when manually running Falco (without using systemd).

ghost · 2021-06-23T12:50:09Z

I didn't have time to perform another test but in the last one on CentOS 7 (#1673 (comment)) I got 2 identical log entries in /var/log/messages

Jun 17 14:55:23 ip-172-31-36-116 falco: 14:55:23.315542901: Error File below /etc opened for writing (user=root user_loginuid=1000 command=touch /etc/domenico parent=bash pcmdline=bash file=/etc/domenico program=touch gparent=sudo ggparent=bash gggparent=sshd container_id=host image=<NA>)
Jun 17 14:55:23 ip-172-31-36-116 falco: 14:55:23.315542901: Error File below /etc opened for writing (user=root user_loginuid=1000 command=touch /etc/domenico parent=bash pcmdline=bash file=/etc/domenico program=touch gparent=sudo ggparent=bash gggparent=sshd container_id=host image=<NA>)

I will try to do some other tests on the last release.

ghost · 2021-07-07T15:29:29Z

With systemd, disabling stdout_output prevent falco from writing in syslog:

stdout_output:
  enabled: false

leogr · 2021-07-07T15:40:52Z

With systemd, disabling stdout_output prevent falco from writing in syslog:
stdout_output:
  enabled: false

Have you tried leaving stdout_output enabled and adding StandardOutput=null to Falco's systemd config file (falco.service)?

ghost · 2021-07-07T16:14:52Z

With systemd, disabling stdout_output prevent falco from writing in syslog:
stdout_output:
  enabled: false
Have you tried leaving stdout_output enabled and adding StandardOutput=null to Falco's systemd config file (falco.service)?

I did not. I will.

ghost · 2021-07-15T08:57:06Z

With systemd, disabling stdout_output prevent falco from writing in syslog:
stdout_output:
  enabled: false
Have you tried leaving stdout_output enabled and adding StandardOutput=null to Falco's systemd config file (falco.service)?

@leogr my colleague did a test and with StandardOutput=null Falco do not write in syslog.

ostendali added the kind/bug label Jun 9, 2021

ghost mentioned this issue Jul 19, 2021

Systemd unit: Do not send falco alerts to /var/log/messages by default #1697

Merged

poiana closed this as completed in #1697 Sep 23, 2021

leogr mentioned this issue Jan 18, 2023

new: ship falcoctl bundled with Falco #2345

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

falco logs contaminating /var/log/messages #1673

falco logs contaminating /var/log/messages #1673

ostendali commented Jun 9, 2021

domenico4sec commented Jun 17, 2021 •

edited

Loading

leodido commented Jun 17, 2021 •

edited

Loading

domenico4sec commented Jun 17, 2021 •

edited

Loading

domenico4sec commented Jun 17, 2021 •

edited

Loading

domenico4sec commented Jun 18, 2021 •

edited

Loading

leogr commented Jun 23, 2021 •

edited

Loading

ghost commented Jun 23, 2021 •

edited by ghost

Loading

ghost commented Jul 7, 2021 •

edited by ghost

Loading

leogr commented Jul 7, 2021

ghost commented Jul 7, 2021

ghost commented Jul 15, 2021

falco logs contaminating /var/log/messages #1673

falco logs contaminating /var/log/messages #1673

Comments

ostendali commented Jun 9, 2021

domenico4sec commented Jun 17, 2021 • edited Loading

leodido commented Jun 17, 2021 • edited Loading

domenico4sec commented Jun 17, 2021 • edited Loading

domenico4sec commented Jun 17, 2021 • edited Loading

Machine

Prepare machine

Installation

Update Falco configuration

Check and start Falco

Reproduce error

domenico4sec commented Jun 18, 2021 • edited Loading

leogr commented Jun 23, 2021 • edited Loading

ghost commented Jun 23, 2021 • edited by ghost Loading

ghost commented Jul 7, 2021 • edited by ghost Loading

leogr commented Jul 7, 2021

ghost commented Jul 7, 2021

ghost commented Jul 15, 2021

domenico4sec commented Jun 17, 2021 •

edited

Loading

leodido commented Jun 17, 2021 •

edited

Loading

domenico4sec commented Jun 17, 2021 •

edited

Loading

domenico4sec commented Jun 17, 2021 •

edited

Loading

domenico4sec commented Jun 18, 2021 •

edited

Loading

leogr commented Jun 23, 2021 •

edited

Loading

ghost commented Jun 23, 2021 •

edited by ghost

Loading

ghost commented Jul 7, 2021 •

edited by ghost

Loading