local.file_match ignore older files

[rveachkc]

Request

Could the alloy local.file_match component be updated with an option to ignore files that have been modified in a certain period of time?

Something like:

local.file_match "tmp" {
  path_targets = [{"__path__" = "/tmp/logs/**/*.log"}]
  ignore_older = 1d
}

Use case

Alloy has run out of memory attempting to scrape log files from a batch job that runs frequently and drops a unique log file per execution.
Alloy has also maintained many more open files than is necessary to run, increasing system resource usage.

While the argument could be made that better log rotation practices should be followed, I feel as if this would be a nice addition to match the feature provided by the filebeat filestream ignore_older feature

[ravishankar15]

Hi Team, I can work on this issue. One of the point to note is if we have this option based on the PollFrequency the export can be inconsistent.

like,

1. sync_period is 10min and File is updated every 11min -> This file will never be included
2. sync_period is 10min File was initially updated every 11min and later started getting updated every 5min -> Initially the file will be excluded later it will get included.
3. sync_period is 10min File was initially updated every 5min and later started getting updated every 10min -> This file will first be included and later it will not be included.

Maintainers note,
I am thinking of passing a the time param to getPaths where we can ignore the files/targets based on the ModTime() on fs.FileInfo. Thoughts ?

[rveachkc]

From my perspective as an end-user of alloy, I believe this behavior is reasonable and should be expected.

The update frequency of a file is just something that needs to be considered when setting this parameter. Perhaps any documentation around it should recommend setting the duration longer than the update frequency of the file.