Option to just read from files (without continuously watching them)

[ppf2]

When using file input against files that are static (will not be appended to anymore), it will be nice to provide an option to exit the LS pipeline once the files have been read (instead of keeping the process running and keep watching for new streams). This will allow the end user to schedule periodic runs to read from files and exit when done.

[jordansissel]

For this it may make sense to have a new input. I wonder if it's too confusing to provide "follow files forever" behavior in the same plugin, and near the same settings, as "read files once, from the beginning" behavior.

[ppf2]

👍

[jordansissel]

Also, a confession from an implementor's perspective, the code to "Find some files, read them until EOF, then be done" is way simpler than the file input is today. :P

[MarkusMayer]

+1 for this feature.
We are importing many (approx 1 Mio/day) static files to ELK (on Windows). The files are written once and never appended. I assume that sincedb/watch is causing a lot of overhead unecessary for such a use case (sincedb dev/nul does not work on windows).

[jordansissel]

I am +1 on this feature. Probably a new plugin for simplicity of interface. I'm not sure what we should call it?

Any ideas? I'm open to making the file input do this as well, but need convincing since this new feature won't need sincedb, or stuff like it that the file input uses today.

[guyboertje]

I am keen to create a simpler plugin called, maybe, logstash-input-read_file. It should reuse most of the current file input but "know" that it is not tailing.

[yehosef]

It's not clear why this needs to be a different plugin. While it's definitely easier than then read-watch case, once you've solved that case it seems adding this logic is not complicated. Somewhere in the code of the current file plugin it's going to say "if EOF wait() and try_again()" - you can just have the a config called eof_action which can do what you want (delete/mv are the basic use cases - you could start there - maybe add run_script to cover everything else).

The advantage of this approach is that it's simpler for me to know that I have one plugin that can do different things - than to keep track of how the file its going to be used and use different plugins to do almost the same thing. It also seems that you'll have to duplicate a lot of the file logic in two plugins instead of providing a pretty trivial hook into the current plugin.

I haven't looked through the code - java/ruby isn't my strong point - but it seems like we shouldn't have to go here.

[guyboertje]

What would your guess be to the percentages of use case for Tail only, Read only and mixed Tail some Read some? My guess is that it may be as much as 49.5, 49.5, 1.

I did start a branch with a mode config of tail, read, but I stopped work on this because most of the heavy lifting in this input is done in the filewatch library. It is heavily optimised for the tail use case.
When we open a file to read it, very occasionally if the timing is perfect we see an empty file - we get an eof then. Obviously this depends on how the user puts the files in the watched folder.
In the read case I would like to support zipped files and breadth or depth first operations. Breadth first means reading 32K from each file in turn until 'done'.

In the tail case, assume for a moment that there are 5000 files detected from the glob and they have all just been rotated and so are empty. The current tail implementation will open all 5000, see that they are empty and loop, we do not open-read-close now. In time, as content is appended we will loop through each file reading what we can and there is no logical eof. We can never consider it 'done'.

In the read case, assume the same number 5000 of content complete files but they are > 800K big. If we reuse the same filewatch lib, we will open each one in turn and begin reading to eof. It will be many minutes before we get to the 5000th file. Further, because filewatch is tailing, we will monitor those 'done' files for changes every stat_interval and the 5000 files will stay open. This is why we are introducing ignore_older and close_older configs and the auto_flush config on the multiline codec but actually they are hacks to help support the read use case. If the user has, say 20 file inputs operating on different folders, when all inputs are opening all of their files, one can run out of file_handles and then many things start going wrong, e.g. the sincedb can't be saved and filters/outputs that use files or sockets begin to fail. I suspect that some of the 'can't recreate' weirdness issues may be related to this.

A specialised readfile input using a read optimised filewatch can do things much more efficiently. We have no need for the stat_interval or start_position. We can have a scan_mode for depth or breath first. We know we only need to stat, open, looped read and close and when 'done' we can 'tidy up' - flush buffered multilines and 'signal' that we are done. We can operate over a smaller bunch of files, say 256, at a time and therefore support reading many many thousands of files in multiple readfile inputs with a low file_handle impact and lower memory usage.

IMO, we would not want a synchronous eof_action that runs a script or operates on a file - we just need to signal to the user that we are done with a file then the user can action this out-of-band - presuming that we can know what 'done' means.

[yehosef]

@guyboertje
Thanks for the explanation - I hear some of the challenges given a combined implementation. Here's my thoughts.

For files that are being added to or rotated, things should work as they do now - as messy as that is. These files won't have an eof_action and will work as now.

There will be other file globs with config that will specify an eof_action. Then will read until the eof and then do whatever the eof says to do. Here, the main problem (I see) is that if I want to be able to add complete files into directory it will take time before it's all copied and I might hit the eof but it's really still copying .Here the stat_interval might be helpful - you might hit the eof, but you would wait until the next stat interval (or two?) to make sure it didn't change. You could also handle this by just changing the filename when finished - file.json.tmp -> file.json - that's basically instant and avoids the problems.. not sure how big of an issue this is.

I think it is important to manage the case where logstash dies or is killed - so I assume you do need a file/position status also - I think it's important that's not overlooked.

It seems that some of the problems related to high number of files will automatically be taken care of - eg, you probably won't have to handle 5000 files because they'll be processed out when finished. You could even support an empty handler to just say - "ignore additional changes" - don't watch - and let other outside scripts handle it.

About the script/action - I agree it shouldn't be blocking - if you don't need to output (which I assume you don't) - you can just fork the process and let it do its thing. But there are many options on how the handle this and what to support - I was just suggesting something easy and flexible - but I am sure there are other/better options.

But all this is really just implementation details - If in the end it is actually better, stronger, faster and able to process large log files in a single bound.. :) and it happens sooner than later - I'm all for it.

[jordansissel]

Somewhere in the code of the current file plugin it's going to say "if EOF wait() and try_again()"

I wish it were this simple, but it's not clear how everyone will be doing these batch file deliveries. For example, if Logstash is running always and watching for new files to process, and a delivery starts of a single file that will eventually be 500GB, Logstash may start seeing that file as new before the first few hundred megs are written, and it is likely that Logstash catches up to the writer, say, by 500MB (in this fictional example), and gets EOF. Now with this approach of EOF meaning "I hit the end of the file" we will miss the remaining 499.5GB of data!

Further, our incentive to make a new plugin is partly because the use cases of files-as-a-stream and files-read-once have pretty different configuration needs. Streams need logrotation detection, file rename handling, etc. Read-once needs EOF actions and won't deal with renames or log rotation -- so if we combined these behaviors into a single plugin, you'd have a bunch of settings that are invalid for entire use cases. For example, "delete the file at EOF" wouldn't be something useful for files-as-streams, since there's no "end" to such streams.

I'm open to having it be one plugin, but not if there are so many mutually-exclusive configuration settings.

[yehosef]

The case you mention where the logstash processing overruns the filesystem copy, I also mentioned. As I said, you can could handle it in logstash by wait for changes after an eof to see if more changes come. If after the stat_interval (or two or three) there are no more changes - that's considered an EOF and whatever is supposed to happen, happens.

I'm curious how you're going to handle this case with the proposed filereader plugin - why will it be simpler - won't you also need to poll and try again?

I looked through the docs on the file plugin and the only option that doesn't make sense for the batch use case is the start_position - you obviously want to start at the beginning. As I mentioned, I think you will still need the sincedb because you need to know where you're up to incase logstash dies. Or even if it doesn't, if the file size is changing because it's being copied in you need to know where you're up to. Which other options are not applicable for the batch process?