WIP: journald reader #1852

sempervictus · 2020-03-05T06:11:03Z

This is absolutely not ready to merge, its scaffold for the badly needed systemd journald reader mechanism. So far all it does, hopefully, is open the journal and start reading messages from the end into our queue. Its late, time's short, so this is very rough and untested so far. Collaboration very welcome.

This seeks to address #1367.

ping @atomicturtle @ddpbsd @dcid

src/Makefile

src/logcollector/read_journald.c

src/logcollector/logcollector.c

ddpbsd · 2020-03-05T12:07:41Z

Thanks for jumping on this!
I'm sure some of the comments I'm leaving are already known issues, but my memory is bad enough that I've learned to leave myself notes.

src/logcollector/read_journald.c

sempervictus · 2020-03-08T17:28:04Z

Nice! It works - was worried merge-squash would lose authorship.
Thanks @illuusio.

sempervictus · 2020-03-09T07:09:48Z

Prototyped a source selector for the journal reader which should allow us to filter which units (now SYSLOG_IDENTIFIERs) we're reading from. This is the "cheapest" way i could think of to do this in a situation where we may have tons of subscribers to individual journals, doing that check and failing 99% of the time. Not sure what that'll do if nothing's passed in, but in my little console version of this it does perform the filtering correctly and writes out the unit name. Reason for strstr vs direct comparison is that systemd can generate auto-named units with common prefixes, and this should help pull from them even if they're added after the reader is initialized.

@ddpbsd: will add hostname too, thanks as always for the useful suggestions.

@atomicturtle, could you please poke at that snprintf to make the message format OSSEC-friendly as a syslog message or whatever format you'd prefer it to have in the queue?

Start process of creating a journald reader using the native systemd interfaces. Make blatant assumption that being on Linux means having systemd, that definition and include directive should be fenced off with a check for the header file being present.

This is a blind journal reader which just reads messages from the current tail-end of the system journal, userspace and kernel, with no consideration for source or position. Next steps: Implement sd_journal_get_cursor and friends for position markers. Implement filtering which journals are read - system, uspace, or specific applications. Implement filtering by string, severity, time, etc. Testing: Absolutely none, this probably won't compile, may eat pets, etc.

* Fix compiling and suspend warnings in read_journald.c * Add READING_JOURNAL #define for debug printing in logcollector * Add 'journald' as allowed XML config parameter for 'logformat'

Use the attribute passed to sd_read_journal as a filter for string comparison with `strstr()` of the _SYSTEMD_UNIT unit field data (after "_SYSTEMD_UNIT=" prefix). This isn't nearly as complete as PCRE/SREGEX matching for units where something like "user\d+@.*" would come into play, but its a lot cheaper, and is really a lazy convenience around having full service/socket/unit names passed to the reader. Next Steps: Severity filtering Cursors/state management Do we want to even think about namespaces? Lots and lots of testing

@ddpbsd

As suggested by @ddpbsd, use SYSLOG_IDENTIFIER to filter for log sources instead of _SYSTEMD_UNIT.

@ddpbsd

Extract the journal entry _HOSTNAME per @ddpbsd, insert into final message string. Clean up message buffers between iterations to avoid matches in error conditions where data from the prior loop iter is used for comparison or output.

USE_SYSTEMD defaults to yes and enabled systemd support on Linux. Compile with USE_SYSTEMD=no to disable it. endif not end. The filter thing didn't work, so only worry if USE_SYSTEMD is 'yes' explicitly. Ignore this file is no HAVE_SYSTEMD Add the systemd-dev pacakge for travis.

sempervictus · 2020-03-09T22:19:44Z

Awesome, looks like Dan fixed the CI.

So who here is comfortable enough with positional cursors on source data to help figure out how we want to deal with keeping track of where we were for each reader before a restart or other feed interrupting event?

sempervictus · 2020-03-12T00:37:18Z

@atomicturtle @ddpbsd @dcid: i think we have hit a small architectural pain point here: our reader and parser code is intertwined, and now that audit logs can come from journald (file or memory) or a flat file (fd), the fact that the audit reader is its own encapsulated file, hard-coded to the fd approach, either means i copy-paste code (same for postgres and others) and make a ton of redundant bloat, or we think about splitting out actual accessors of API/ABI endpoints and file descriptors (the literal "readers") from the parsing and formatting code.
Thoughts? Interns to sacrifice on the altar of clean-code?

illuusio · 2020-03-12T19:45:21Z

src/logcollector/logcollector.c

+        else if (strcmp(logff[i].logformat, "journald") == 0) {
+#ifdef HAVE_SYSTEMD
+            verbose(READING_JOURNAL, ARGV0, logff[i].file);
+            sd_read_journal(logff[i].file);


Just wondering is this correct way of handling this as other reader just make logff[i].read = sd_read_something(with args) and journald reader hangs forever?

illuusio · 2020-03-12T19:46:47Z

src/logcollector/read_journald.c

+      );
+      // Strip off "SYSLOG_IDENTIFIER=" prefix for unit comparison inline
+      if (strstr((char *)(jsrc + 18), unit) == (char *)(jsrc + 18) ) {
+        // Read hostname


There should be a way to log all journald stuff as reading /var/log/messages as this just logs some units output.

ddpbsd · 2020-03-15T20:00:08Z

@sempervictus It depends on how quickly you want journald support. Adding another reader in the same way previous ones were added should be eaasier than refactoring all/most of them.

* If unit name is same as config unit then log message * If config unit name is all then log everything (same as syslog reader) * If config unit name starts with char '/' then assume one wants all but can't configure it correctly

… Journald is optional (#6)

sempervictus · 2020-03-31T19:41:07Z

@ddpbsd: makes sense.
@illuusio: thanks for all the commits, merged.
@ALL: so cursors are the remaining item on the list - we need to be able to pick up where we left off during an agent restart, ideally.

illuusio · 2020-04-01T05:33:06Z

@sempervictus only thing with cursor is where to store them? If used multiple inputs is there some commong way in OSSEC-HIDS to store such info if there is not we shoud re-invent wheel with JSON or such?

* Use Syslog-ng time string to pass ossec-analysisd time parsing functions. After that message is corretly separated in parts and rules behave corretly * Add needed ':' to have program_name parsed as well

) * Add new variable 'void *ptr' to struct logreader to handle situations like Journald where there is need to keep track of place with something diffrent than 'FILE *'. Currently only user is journald but in future it could be used by other readers * Change Journald reader behave as every else reader out there. This means that reading loop is handling looping not reader itself. This meas there can be other readers as well with Journald-reader

illuusio · 2020-04-06T12:45:37Z

Now this should be Beta quality as I got it crashing couple time but can we get some testers #1367 (I have been testing this on couple servers and it work quit nice).

sempervictus · 2020-04-06T14:07:38Z

@illuusio: what're you using for your decoders and rules to see data, or are you seeing data hit the manager?

illuusio · 2020-04-06T14:25:38Z

@sempervictus I just hardcoded version that printed on analysid inner things. Servers sends me emails about something is wrong. I have to investigate more to see if rules really work as they should but messages are parsed (and separated) as they are coming from syslog-ng (which it is mimicking).

illuusio · 2020-06-03T08:44:40Z

I've been testing this while now in productions and it works fairly well. It needs to have some rules for journald but I think this could be merged and rest of the problems solved after it receives more testing from community.

sempervictus · 2020-06-18T22:42:18Z

Yeah, i'm good with pushing this into master and wider testing.
One remaining through comes to mind: do we want to permit the reader to restrict which hostname it reads from? There are two competing and completely valid use cases here:

An OSSEC-equipped host logging all of its dockers or what not
An OSSEC-equipped host with OSSEC-equipped LXD/LXC containers. For all we know, they have separate OSSEC masters, and only want their own data.

sempervictus · 2020-06-18T22:44:36Z

@illuusio: this effort is going to be the subject of a talk or two, any objections to me giving you a shout-out for all the hard work by the name listed in your GH profile? Any prefered alternative for given name or handle than the ones listed there?

illuusio · 2020-06-19T05:57:35Z

1. An OSSEC-equipped host logging all of its dockers or what not

2. An OSSEC-equipped host with OSSEC-equipped LXD/LXC containers. For all we know, they have separate OSSEC masters, and only want their own data.

Is there such restrictions from other readers? I'm just asking as this kind of a easy task but should be done as 'OSSEC'-way if there is any and is there some -tag in config for this?

Any prefered alternative for given name or handle than the ones listed there?

Nope just use that if you prefer to give credit but is not necessary as part of security in your life is to play low 😸

sempervictus · 2020-06-24T11:33:26Z

@illuusio: There are no such restrictions from other readers AFAIK, but other readers access inodes which are inside the namespace of the container, whereas the host's journal will have entries from journald-aware child containers.
First talk on this is today @ 1530 EST in the OSSEC virtual webinar (COVID replacement for the normal annual conference)

illuusio · 2020-06-24T12:57:25Z

Hopefully there is some video as it's kinda late for me. So If I understood correctly the problem is that you have bunch of containers which report to one server and there should be restricting tag to make it choose which it allows to take input like:

<allow-host>megahost</allow-host>
<allow-host>container-host1</allow-host>

Or I'm missing something.

sempervictus · 2020-06-24T13:02:24Z

@illuusio: we'll have video, either atomicorp will have a copy or both they and i will.
Since the configurations have to apply to all OSSEC agents, i'd say we do something like:

<journal-namespace>local</journal-namespace>

or

<journal-namespace>all</journal-namespace>

and then have it pull the current NS' hostname to compare

illuusio · 2020-06-24T13:07:25Z

Ok and local only reports local stuff and all everything. I take a look as I have to get OSSEC working with agents soon and need this one be number one citizen.

ddpbsd · 2020-06-24T18:01:22Z

Please add libsystemd-dev to the list of required packages in INSTALL.

sempervictus · 2020-06-24T22:21:01Z

@illuusio: thank you. Based on the conference today, sounds like we want to be able to filter host/host+containers, which log source (application/service), and even potentially a substring match in the event output, though i think that can be added later - we just wanna hit master branch with a stable configuration API (in terms of not changing parameters down the line, only potentially adding some if people want more filtering capacity @ the reader).
@ddpbsd: good catch! wouldn't have caught that since Arch has the development libraries right in the main -libs package

atomicturtle · 2021-01-06T18:53:37Z

merging this in so we can start more open testing

sempervictus force-pushed the feature/reader_journald branch from 2fb9c8b to 0f0a9aa Compare March 5, 2020 06:13

sempervictus mentioned this pull request Mar 5, 2020

Are there any plans to support journald logs? #1367

Closed

sempervictus force-pushed the feature/reader_journald branch 2 times, most recently from 5fd14b0 to 5828876 Compare March 5, 2020 06:30