WIP POC Event categorization on Linux auth logs

[webmat]

To me, it's looking like the current Ingest Node processors make this task much too verbose.

General TODO

• Add more kinds of auth logs (PAM, su, systemd)
• Refactor syslog header parsing:
  • Now defined once, instead of in each pattern
  • Now always extracts process.name and .pid

Categorization TODO

• All activity: event.kind:event
• SSH authentication activity: event.category:authentication, event.action:ssh_login,
  event.outcome:(success|failure)
• SSH session activity: event.category:session,
  event.action:ssh_session, event.outcome:(connect|disconnect)
• SSH management activity (e.g. restarts): event.category:?, event.action:?
• User & group management activity
• sudo activity
• su activity
• PAM activity

[elasticmachine]

Pinging @elastic/secops

[webmat]

Based on the work in this proof of concept, we'll need a much more terse way to define these rules:

• developer sanity: very few logs are categorized right now, and this has already required a lot of brittle code to be put in place
• Ingest Node performance limitations: all of the if clauses are in Painless, and will count towards increasing the likelihood of causing script.max_compilations_rate errors.
  • We're already facing problems in CI with one Painless script and perhaps 1-2 if clauses per module. Going through with this approach will increase the Painless script count by more than an order of magnitude, I think. This will not be viable for CI, much less for production environments.
  • See Parameterizing Painless script #9821 for more information about the script.max_compilations_rate errors.

Some things that could help:

• An if processor that takes an array of processor tasks (same semantics as the array under "processors":)
• The ability to set multiple field values, when a grok is successful. Similar semantics to Logstash' add_field common option

[tsg]

Thanks for putting this together @webmat.

Based on the work in this proof of concept, we'll need a much more terse way to define these rules

Looking at the final pipeline, it looks reasonably short and actually quite readable.

There are for sure ways to improve the developer experience, and we should brainstorm about them, but based on the above I wouldn't say this is unfeasible in the current form. Could you explain perhaps a bit more what you found to be terrible?

Regarding the max_compilations_rate, we should discuss with the ES team if it might make sense to raise that soft limit for everyone in 7.0, but we can also have Beats raise it per cluster during setup, or something similar. It also feels like something more likely to affect CI rather than production, where every script is only compiled once. CC @jakelandis.

[webmat]

@tsg The reason I say it's not reasonable is that so far this PR so far is only doing categorization on 1/7 types of messages that I see in this module (there may be more than 7). Moreover, it's not doing so properly, as it incorrectly makes the assumption that anything SSH is about authentication attempts. But SSH messages can also contain admin stuff (restarts) and sessions (connects / disconnects). Given all this, I think the amount of code necessary is prohibitive *.

Another part that bugs and makes things brittle me is how each processor has to replicate a full if statement with a lot of duplicate checks (repeating other processor's similar if).

I'll work on the PR some more this week, to continue fleshing out the POC. I think it will better showcase the problems I see.

Note that we could take a completely different approach as well, and go with one single bigger Painless script. Multiline, indentation, etc. But we don't have good support for this in Beats yet.

* I did start this PR by doing a pretty good refactoring that extracts out the parsing of the syslog header. It used to be done by each grok pattern. Even if we end up dropping this POC, I'll extract the refactoring and submit it on its own. This refactoring is working against me making the case that this generates too much code, since I started with a code reduction ;-)

[webmat]

When evaluating this code, make sure you scroll right. Check out line 71 ;-)

[webmat]

Another note that's unrelated to Ingest Node verbosity is that we'll need to do de-duplication as well. In many cases, a single real world event (e.g. successful login) can translate to 3-4 log entries or more. See this successful login. We'll need to de-duplicate this, to reduce noise.

This won't be done with Ingest Node, but in the current plan, I'm not sure what would do this.

This comment is more of a note for later than a problem to be addressed in this POC, though.

[jakelandis]

Regarding the max_compilations_rate, we should discuss with the ES team if it might make sense to raise that soft limit for everyone in 7.0

I think the real is issue elastic/elasticsearch#37120. I doubt (but haven't looked) that you actually have that 75 scripts/templates that need to compile. I will look into getting that issue resolved very soon. I would prefer to not raise the limit unless we know we the limit will be hit after elastic/elasticsearch#37120 is resolved.

[webmat]

@jakelandis Ok, thanks for bringing this one up. We've bumped the Beats CI pipeline to 1000 compilations / minute for now 😆

I'll make sure nobody tries to "fix" any of the script and let them know about this issue