Promtail about rotated log

[pourer]

In my scenario, __path__ congfig as /usr/local/test/*.log, and log file named test.log.
When the log file test.log reached 100M, it will be renamed to test.20200707-183213.408507.log and start a new test.log.
Now, promtail will start new tailer for test.20200707-183213.408507.log with SeekInfo as {Offset:0 Whence:0}, but test.20200707-183213.408507.log scraped before it rotated.
Is this not a problem?

[owen-d]

You'll need to make sure that the rotated logs are not monitored by promtail as it has no way of knowing if a rotated file is "new" or not. This can be done multiple ways, including by compressing logs during rotation (test.log.gz wouldn't match your config) or by changing where they're rotated to, ensuring that they no longer match the glob path promtail is monitoring.

[slim-bean]

I had a couple thoughts but couldn't make them work which is a bit disappointing, first I was hoping a relabel_configs section might let us do some regex matching but the glob is not expanded before the relabel configs are applied so this didn't work.

Second the awesome bmatcuk/doublestar library we use for the glob matching has some support for character class matching but none of my attempts to match this kind of pattern were successful.

I think for the moment you are stuck with separate configs for every log you want to match and match /usr/local/test/test.log directly so that you won't hit the rotated files.

Or like Owen suggested, you need a way to rotate the files such that they no longer match the glob pattern, often i see logrotate.d do this by appending the rotation number after the extension: test.log.1

I'm gonna leave this issue open though because i'm wondering if it's possible to do the glob expansion before we apply the relabel configs but don't have time to investigate this right now.

[pourer]

You'll need to make sure that the rotated logs are not monitored by promtail as it has no way of knowing if a rotated file is "new" or not. This can be done multiple ways, including by compressing logs during rotation (test.log.gz wouldn't match your config) or by changing where they're rotated to, ensuring that they no longer match the glob path promtail is monitoring.

Hi @owen-d , thanks for your quick response. I got it.

[pourer]

I had a couple thoughts but couldn't make them work which is a bit disappointing, first I was hoping a relabel_configs section might let us do some regex matching but the glob is not expanded before the relabel configs are applied so this didn't work.

Second the awesome bmatcuk/doublestar library we use for the glob matching has some support for character class matching but none of my attempts to match this kind of pattern were successful.

I think for the moment you are stuck with separate configs for every log you want to match and match /usr/local/test/test.log directly so that you won't hit the rotated files.

Or like Owen suggested, you need a way to rotate the files such that they no longer match the glob pattern, often i see logrotate.d do this by appending the rotation number after the extension: test.log.1

I'm gonna leave this issue open though because i'm wondering if it's possible to do the glob expansion before we apply the relabel configs but don't have time to investigate this right now.

@slim-bean Okay, thank you for sharing all your thoughts. I was wondering if we can take tailer with inode info to solve my scenario?

[fr6nco]

Hi,

just reopening this discusson. Looking into replacing heavy ELK stack with loki and filebeat with promtail.

How do you ensure that all the content of the file is read on rotation?
Elastic stores the Inode of the file with the offset descriptor. On rotation the rotated file is picked up and based on the inode is matched against a descriptor and finishes shipping of lines of the rotated files.

In promtail, you should exclude rotated files from the pattern. When a log file is rotated and lets say promtail didnt manage to ship all the logs prior to rotation, how do you ensure, that the logs added to the logfile before the last scape and the rotation are shipped to loki?

Thanks

Some info on rotation for filebeat:
https://www.elastic.co/guide/en/beats/filebeat/current/file-log-rotation.html

[pourer]

Hi,

just reopening this discusson. Looking into replacing heavy ELK stack with loki and filebeat with promtail.

How do you ensure that all the content of the file is read on rotation? Elastic stores the Inode of the file with the offset descriptor. On rotation the rotated file is picked up and based on the inode is matched against a descriptor and finishes shipping of lines of the rotated files.

In promtail, you should exclude rotated files from the pattern. When a log file is rotated and lets say promtail didnt manage to ship all the logs prior to rotation, how do you ensure, that the logs added to the logfile before the last scape and the rotation are shipped to loki?

Thanks

Some info on rotation for filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/file-log-rotation.html

@fr6nco This is indeed a problem. So, if we can take tailer with inode info, this problem can be solved. However, I'm not sure if promtail already supports this solution now.

[slim-bean]

How do you ensure that all the content of the file is read on rotation?

Promtail keeps a handle to the open file from which it is currently reading, when a rotation event is detected, it will read the remaining bytes of the current file before moving to the new file.

[frittentheke]

How do you ensure that all the content of the file is read on rotation?

Promtail keeps a handle to the open file from which it is currently reading, when a rotation event is detected, it will read the remaining bytes of the current file before moving to the new file.

@slim-bean this holds true only if Promtail is running with the file handle open during the rotation. If it was not, there is no way to reconcile and data is lost (as in "not shipped" to Loki). There really seems no way around introducing the inode to positions to "abstract" away from the name of the file. The name simply changes for most cases / implementations of log rotation.

There also is a more recent issue about this exact issue: #6503