RPM package: Do not log into systemd journal

[spinscale]

When the RPM package is installed and elasticsearch is started via systemd, every log message is logged in /var/log/elasticsearch/ and to the systemd journal, which can be accessed using the journalctl tool.

I dont think we should put anything into the journal, especially as most other services seem to be kinda scarce on output in there (given my vagrant vm).

Open for discussion.

[dakrone]

@spinscale why not? Storing the logs in the journal allows an administrator to do something like: journalctl _PID=1234 --since "20 min ago" that is more difficult to do tailing logs.

Is there any reason why shouldn't put logs in it on systemd systems?

[spinscale]

As far as my knowledge about systemd journal goes, by default it cleans itself up in case it takes up too much space. If Elasticsearch is on a log blast, we fill up the disk twice (which I dont care too much about, monitoring will kick in and notify), but this might expire other important entries out of the journal logs, where we have the es logs anyway.

That said, I do like the querying capabilties :)

[dakrone]

If Elasticsearch is on a log blast, we fill up the disk twice (which I dont care too much about, monitoring will kick in and notify), but this might expire other important entries out of the journal logs, where we have the es logs anyway.

That sounds like evidence that we should log only to the journal on systemd machines, since you can easily configure the journal to write to a log file additionally.

[nik9000]

So long as we only log one place I really don't care which place that is.

[jasontedor]

I'm in favor of systemd logging, but if we are only going to log to one place I think that it should be /var/log/elasticsearch/elasticsearch.log; the journal can be confusing, and has caused trouble for users in the past (because they don't know to look there, and the vast body of knowledge accessible through searches will point people towards /var/log/elasticsearch/elasticsearch.log rather than systemd). See #16159 and the related issues for some recent issues that support this.

We can provide commented out journal configuration for end-users that want this instead of the simpler default.

[tlrx]

When it comes to elasticsearch as a system service I think we should try to integrate as much as possible with the OS features. I feel like systemd is adopted by more and more distributions and the journalctl is really powerful. We could enable persistent log with a max file size and compress by default.

Concerning the vast body of knowledge referring to /var/log/elasticsearch/elasticsearch.log file, I think dropping a file there with a single line comment saying to look at the journal is OK.

[jasontedor]

@tlrx I'm all in favor in systemd but my argument is merely that if we are going to pick one, we should pick the simplest. As an added note, on all other systems we still have to support /var/log/elasticsearch/elasticsearch.log.

We could enable persistent log with a max file size and compress by default.

We can also provide a logrotate.d conf file for this.

Concerning the vast body of knowledge referring to /var/log/elasticsearch/elasticsearch.log file, I think dropping a file there with a single line comment saying to look at the journal is OK.

That does not alleviate my concerns that I've encountered many end users that have found the journal confusing.

[spinscale]

I also completely forgot we are still bound to maintain init.d based systems, due to LTS releases from various distributions.

Proposal from my side, unless we are on systemd exclusively (lets see if that works out without another dozen of init systems jumping into existence): Remove the console from logging.yml, but also add documentation how to do systemd logging exclusively and include some of the nice command line arguments that Lee showed off above and mentioning to not care about log rotatation in that case.

[clintongormley]

Discussed in FixitFriday, agreed with #17220 (comment),, especially given that having logs in a standard place makes it easier for our diagnostic tools to capture them