Meta Issue to track outstanding changes to the file input

[guyboertje]

Items on this list may be covered by one or more existing issues, I will circle back and update the list when I spot the issue.

• Add "feeder" support to DIscoverer, for when the discovered file set is so large that file reading is delayed significantly.
• Allow the file path to be used as the sincedb key. Used as an alternative to inodes when the file path is known to be a unique identifier (more strongly than the inode combo anyway) e.g. /dir/<content_type>-1536495948-769372.log (NFS).
• Handle errors other than ENOENT that effectively mean the same thing - the file has gone (NFS).
• Allow read mode to detect files still growing, e.g. being "filled" over a slow remote link (NFS).
• Allow read mode to detect file rotation.
• Align clean_sincedb_after, close_older and ignore_older with Filebeat's settings, adding new ones as needed.
• Add "move" file completed action and a file_completed_move_path setting.
• Remove sincedb entry on successful deletion of the file, if using inodes as identifier only.
• Add delete_after to read mode to delay the delete (might be covered by syncing with filebeat ^)
• Add support to bypass the delimiter boundary detection - send chunks instead of lines, needs label on the codec to say it does boundary detection.
• Add support to send the byte offset of the chunk or line and file size (read mode) - needs additional work at the codec level.
• Log at WARN when number of files discovered is zero.
• Add support to detect when a filesystem has been unmounted err: Errno::EAGAIN and spin until files are seen again.
• In register, change ArgumentError to ConfigurationError.
• Check file permissions in Discoverer for each file.
• Rotated files should always start from the beginning (i.e. ignore start_position)
• Option to exit pipeline when read mode has reached the end of the file (Option to terminate pipeline when EOF is reached #212).

[ganeshsi]

Its been more than a few months for those listed above , we are looking at a few of the things to be used in our ls pipelines, can we know if any of these are. being worked upon?

[guyboertje]

@ganeshsi
Sorry, I'm not working on the file input at present. The LS team at Elastic is small, 6 (7 soon) people and we are spread across the main LS repos and about 80-100 plugins.

[ganeshsi]

@guyboertje thanks for the response !

[guyboertje]

Out of interest, which changes do you (or any other reader) consider most needed? It would help if you could rank them.

[ganeshsi]

Two of these infact

• Add "feeder" support to DIscoverer, for when the discovered file set is so large that file reading is delayed significantly.
• And logstash stuck in read mode on big number of files #219
  We've worked around with an internal change for latter by adding a "no" option for the sort bits
  We use logstash to manage filesystem sources having significant number of files

[jbwl]

@guyboertje Most needed by far is: Option to exit pipeline when read mode has reached the end of the file (#212).

[MikeSaveItiviti]

I am interested in support to send the byte offset of the chunk or line and file size (read mode) so I don't have to use filebeat in the equation.