tail_files behavior #2613
Comments
|
During startup of filbeat 5.x, a first scan is done to set all the states. During this scan |
|
@jjfalling One question that popped up during implementation: Should only the files be tailed for which no state exists yet or all of them? Means if a file was harvested before and there is already a state of an offset point to the middle of the file, should filebeat set the offset to the end in the first scan or because a state exist start sending the files from the middle of the file? |
Previously if a file was falling under ignore_older on startup and no previous state was found, also no state was persisted. This was important in 1.x releases to prevent the registry file to contain too many files as no cleanup was possible. With the new clean_* options the registry file can now be cleaned up. For consistency all files that fall under ignore_older should have a state, even if they are not crawled before. This change is also required to move tail_files under the prospector (elastic#2613). Otherwise files which fall under `ignore_older` the state could not be set to the end of the file.
Previously if a file was falling under ignore_older on startup and no previous state was found, also no state was persisted. This was important in 1.x releases to prevent the registry file to contain too many files as no cleanup was possible. With the new clean_* options the registry file can now be cleaned up. For consistency all files that fall under ignore_older should have a state, even if they are not crawled before. This change is also required to move tail_files under the prospector (#2613). Otherwise files which fall under `ignore_older` the state could not be set to the end of the file.
|
Checking the behaviour of nxlog:
In summary, nxlog does not apply |
tail_files is now only applied on the first run and after that ignored. Also the state for all files falling under tail_files and not having a state, a state will directly be written. * Implement tail_files by setting ignore_older to 1ns for the first run * Fix typo in stats variable names Closes elastic#2613 and elastic#2788
tail_files is now only applied on the first run and after that ignored. Also the state for all files falling under tail_files and not having a state, a state will directly be written. * Implement tail_files by setting ignore_older to 1ns for the first run * Fix typo in stats variable names Closes #2613 and #2788
Most users might expect that tail_files would apply
ReadFromLastonly on the first scan and from then on it is not taken into account anymore. This behavior was not possible in 1.x because of filebeat worked but it should be possible to change it in the 5.x versions.The text was updated successfully, but these errors were encountered: