Filebeat 5.0.0-beta1 is failing to ship a portion of a file.

[ckmcd]

• Version: 5.0.0-beta1
• Operating System: Linux XXXXX 2.6.32-573.12.1.el6.x86_64 I can't install packetbeat #1 SMP Mon Nov 23 12:55:32 EST 2015 x86_64 x86_64 x86_64 GNU/Linu

I discovered an issue where I had missing log messages in Elasticsearch. I found the missing messages in a local log file. I found the file in question in the registry and discovered that the file had not been completely shipped. i.e. the registry offset was less than the file's size.

Here is my config

filebeat:
  prospectors:
    -
      paths:
        - /opt/mount1/sa-stats/log/local/fast/*.log
        encoding: plain
      input_type: log
      close_removed: true
      clean_removed: true
      document_type: log4j-itg
      multiline:
        pattern: "^[0-9]{4}-[0-9]{2}-[0-9]{2}T([0-9]{2}:){2}[0-9]{2},[0-9]"
        negate: true
        match: before

Note these files are being rotated (renamed) by log4j so we are seeing Filebeat logs like.

/usr/local/st/filebeat5/logs/filebeat.log.3:2016-09-30T13:01:36Z INFO File is inactive: /opt/mount1/sa-stats/log/local/fast/fast.log. Closing because close_inactive of 5m0s reached.
/usr/local/st/filebeat5/logs/filebeat.log.3:2016-09-30T15:50:04Z INFO File is inactive: /opt/mount1/sa-stats/log/local/fast/fast.log. Closing because close_inactive of 5m0s reached.
/usr/local/st/filebeat5/logs/filebeat.log.3:2016-09-30T15:55:43Z INFO File is inactive: /opt/mount1/sa-stats/log/local/fast/fast.log. Closing because close_inactive of 5m0s reached.
/usr/local/st/filebeat5/logs/filebeat.log.3:2016-09-30T16:06:23Z INFO File is inactive: /opt/mount1/sa-stats/log/local/fast/fast.log. Closing because close_inactive of 5m0s reached.

The weird thing is that I see no new harvesters being started.

Restarting Filebeat caused new harvesters to be created for the renamed files and they were entirely shipped.

• Steps to Reproduce:

[ruflin]

Few questions:

• Did you check by coincidence to what line the offset belonged when it stopped?
• Any chance to share some of the log files where you saw the issue?
• How often did you see this issue? Just once or did it happen to multiple files?
• Can you share the full log file?
• Did you see any errors in the log file?
• What is the output your are using?
• Are you using a share volume to read the files from?

As the above files are already rotated 2-3 times I kind of expect that the reading finished and no new harvesters must be started. Or can it happen that after 5 minutes still data can be added to rotated files?

[ckmcd]

Few questions:

Did you check by coincidence to what line the offset belonged when it stopped?

No sorry.

Any chance to share some of the log files where you saw the issue?
How often did you see this issue? Just once or did it happen to multiple files?

Multiple files

Can you share the full log file?
Did you see any errors in the log file?

I assume you mean the logbeat logfile? No I did not

What is the output your are using?

logstash

Are you using a share volume to read the files from?

No

As the above files are already rotated 2-3 times I kind of expect that the reading finished and no new harvesters must be started. Or can it happen that after 5 minutes still data can be added to rotated files?

Data is not added to the files after rotation. I think you are right and that the problem is that no new harvester is started for the file.

Sorry I can not provide you with everything you asked. For if I see it again I will do a better job collecting data.

[ruflin]

Thanks for all the answers. It would be really great if you could provide some filebeat log output next time this happens. Or if you manage to reproduce this in some way, the steps to reproduce it would be also very useful.

[ph]

I think we can close this, it's really old.