Watch new files in a directory, in an efficient way

[dooblem]

From: https://discuss.elastic.co/t/can-filebeat-watch-for-new-files-created-in-a-directory/57811/3

Magento is a PHP ecommerce application. In case of severe problems, Magento crashes, generating a file for each crash in a var/report directory.
The file contains informations about the crash (db unreachable for instance). The file name is a random number.

I'm looking for a solution to detect and send those reports automatically when they appear in the var/report directory.

I tried using the directory as a path : not working:
prospector.go:293: Skipping directory: /var/www/magento/var/report/

For now I will simulate a log file with a bash script loop running in background, but this is not very handy.

I tried @magnusbaeck solution of using a wildcard, however my registry file is growing indefinitely... And I do not want to disable registry because I have other log files to watch.

I imagine the following problems need to be addressed:

• find a way to record the state of what has been sent (for now, in my bash script workaround, I'm recording the last file sent, and sending any file newer). A new input_type?
• Is the wildcard method efficient if there is thousands files in the directory (/var/www/magento/var/report/*)
• ("cherry on the cake") provide a way to send the file modification time along with the event

Thanks in advance,
Marc

[ruflin]

• Growing registry: The constantly growing registry file is fixed in the 5.0-alpha releases with the clean_* variables. See Registry state persistence and cleanup #1600 You can download it here: https://www.elastic.co/downloads/beats/filebeat
• I think the points for last file sent are more or less already solved in filebeat. As your files only get updated once you can use the new close_eof config options and your files are closed directly after reading: https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#_close_eof
• It would be interesting to hear if you hit a performance issue with *. To check which files have a newer modification date, we would probably also need to check every single file?
• For the file meta data there is an other issue here: Filebeat meta data fields #1775

One issue I can see that you would potentially hit is number of open files on startup: #2236 In case you are only interested in the most recent events, a low ignore_older value could help.

It would be great if you could try out 5.0.0-alpha5 and let us know if you hit any issues.

[dooblem]

Thanks a lot.
Yes,clean_* and close_eof would do it !

Performance: yes, my bash script basically use the find -newer command, going through all the files, but not loading the whole list in memory (I think the wildcard will load everything in memory).
Anyway, we will clean the directory in order to keep a reasonable number of files, to work around that. I'll let you know.

I asked my hoster to install the new version. Unfortunately I may not be able to test it right now.

[ruflin]

No sure if there is something similar to find -newer in golang, especially something that is cross platform compatible.

Please let us now when you have tested further. I suggest we close this issue as I think all the follow up feature requests need currently are already in other issues. If the wildcard causes some memory issue, it is probably best to open a separate PR. Feel free to post your results / findings here even though it is close. Reopen if you think it is needed.