Getting frequent warnings about inbound HTTP traffic rule

[carestad]

After a recent reboot of a server, I am seeing quite frequent warnings from the "Inbount network traffic to HTTP server on unexpected port" rule.

The warning just states the connection (fd.name) is 0.0.0.0:0 -> 0.0.0.0:0. When adding proc.cmdline and proc.pcmdline to the output, it states that both are apache2 -k start.

Is this something others are experiencing as well? Could it be a bug with Falco, or is it something weird going on with my Apache setup?

[mstemm]

I enabled the rule HTTP server unexpected network inbound traffic from /etc/falco/rules.d/application_rules.yaml (I'm pretty sure that's the one you're using?) and restarted apache a few times without noticing any events.

I wonder if you're experiencing dropped system calls, which can result in incomplete state within falco and as a result false positives. Can you try running falco with -v? At shutdown, you can look at the Driver Events: and Driver Drops: values. You can also try -s <stats file> to write periodic stats on events received and dropped to a file, and then check the file to see if you have a non-zero drops count.

[carestad]

I enabled the rule HTTP server unexpected network inbound traffic from /etc/falco/rules.d/application_rules.yaml (I'm pretty sure that's the one you're using?) and restarted apache a few times without noticing any events.

Yes, that is the one.

We have a few other servers used for testing etc. as well, running with the same setup, but this has not been reported from any of them. I would assume it is also connected to the load on the production servers.

I wonder if you're experiencing dropped system calls, which can result in incomplete state within falco and as a result false positives. Can you try running falco with -v? At shutdown, you can look at the Driver Events: and Driver Drops: values. You can also try -s <stats file> to write periodic stats on events received and dropped to a file, and then check the file to see if you have a non-zero drops count.

That sounds more like it. I can give it a try and fire up Falco with both -v and -s <file> though and see what it reports 🙂

[carestad]

Gotten two messages of the kind so far today. The neither output from -v or the file set with -s reports anything unusual when this occurs.

[carestad]

After running falco with -v -s /tmp/falco-stats.log for about 5-6 days now I can report that the verbosity does not help much, in fact nothing at all.

The stats file (/tmp/falco-stats.log) that is written to I can not really make too much sense out of, but I have found that in the JSON, only the first sample property, the cur.events property and delte.events property really change. The cur.drops isn't 0 but also does not change (stable at 1174047).

cur.events seems to increment in general as if it was a timestamp, sample seems to increment per line and delte.events seem quite random. Not sure if any of that is of any help at all.

I will try and upgrade to 0.13.0 and see if that changes anything.

[mstemm]

Thanks for that info. cur.drops is the most important one. It shows on an interval-by-interval basis how many dropped system calls there were. And if that value is mostly 0 and is 0 when you saw the falco alerts, then dropped system calls is probably not the source of the problem you're seeing.

From looking more closely at the inbound macro, I think a better version of it could be the following:

- macro: inbound
  condition: >
    (((evt.type in (accept,listen) and evt.dir=<) or
      (evt.type in (recvfrom,recvmsg) and evt.dir=< and
       fd.l4proto != tcp and fd.connected=false and fd.name_changed=true)) and
     (fd.typechar = 4 or fd.typechar = 6) and
     (fd.ip != "0.0.0.0" and fd.net != "127.0.0.0/8") and
     (evt.rawres >= 0 or evt.res = EINPROGRESS))

This handles a few additional ways messages could be received (recvfrom,recvmsg) but also explicitly ignores non-blocking accepts in a better way than the old macro.

Do you want to try that version of the inbound macro instead? The changes are also in this PR if you'd just like to take the whole rules file: #470.

[carestad]

Applied this last night and so far so good. Usually there are a couple of messages during the morning "rush traffic" on the sites, but nothing so far.

[mstemm]

Great, let us know how it looks over the weekend.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.