[filebeat] autodiscover remove input after corresponding service restart

[seagiv]

Filebeat: 7.2.0 docker container
OS: Ubuntu 18.04.2 LTS
Parity: docker container

config filebeat.yml:
filebeat.txt

1. started filebeat, parity events appeared in kibana
2. stopped/started parity: docker stop parity, docker start parity
3. on first attempt everything went fine, but on second attempt like 10 sec after parity started got number of messages like
   2019-07-18T06:34:14.349Z DEBUG [autodiscover] autodiscover/autodiscover.go:230 Got a stop event: map[config:[0xc0002290b0] container:{"id":"44e30ae98d153681ab77c05346100a78278c39b266e965ae8d0119891194b05a","image":{"name":"parity/parity:v2.4.5"},"labels":{"MAINTAINER":"Parity Technologies \u003cdevops-team@parity.io\u003e"},"name":"parity"} docker:{"container":{"id":"44e30ae98d153681ab77c05346100a78278c39b266e965ae8d0119891194b05a","image":"parity/parity:v2.4.5","labels":{"MAINTAINER":"Parity Technologies \u003cdevops-team@parity.io\u003e"},"name":"parity"}} host:172.17.0.3 id:44e30ae98d153681ab77c05346100a78278c39b266e965ae8d0119891194b05a meta:{"container":{"id":"44e30ae98d153681ab77c05346100a78278c39b266e965ae8d0119891194b05a","image":{"name":"parity/parity:v2.4.5"},"name":"parity"},"docker":{"container":{"labels":{"MAINTAINER":"Parity Technologies \u003cdevops-team@parity.io\u003e"}}}} port:8545 provider:7614a03c-df13-4169-8cbe-1afceb242df3 stop:true]

and filebeat stop processing parity logs despite the fact parity is running and healthy and its logs are ok

it looks like filebeat process stop event after parity start and think container is down

here is full log of this:
filebeat_parity.log

Steps to reproduce:

1. configure filebeat to autodiscover parity docker container
2. stop/start parity
3. wait few seconds and sometime (very often) it stop processing parity container log after receiving stop event despite the fact parity started and running

[jsoriano]

@seagiv thanks for reporting this, I think that there can be actually an issue if the same container is stopped and then started, as in both cases the container will have the same id, that is used to decide if a module has to be stopped or started.

One thing you can try to do to mitigate this problem is to remove the delay to stop the module, this can be done by setting the cleanup_timeout option to zero:

filebeat.autodiscover:
  providers:
    - type: docker
      cleanup_timeout: 0
        ...

[seagiv]

@jsoriano thanks, this fixed the issue, but there is another issue appeared after fix: for around 10m after container restart filbeat log getting spammed with

2019-07-19T06:24:00.426Z ERROR [autodiscover] cfgfile/list.go:96 Error creating runner from config: Can only start an input when all related stat
es are finished: {Id:778937-66305 Finished:false Fileinfo:0xc0009841a0 Source:/var/lib/docker/containers/44e30ae98d153681ab77c05346100a78278c39b266e965ae8d01198
91194b05a/44e30ae98d153681ab77c05346100a78278c39b266e965ae8d0119891194b05a-json.log Offset:1489 Timestamp:2019-07-19 06:23:44.122288114 +0000 UTC m=+761.9208127
36 TTL:-1ns Type:docker Meta:map[] FileStateOS:778937-66305}

and not processing logs during that, after around 10m it starts processing it again

[jsoriano]

@seagiv thanks, we'll have to take a look to this case

[jsoriano]

Second issue is #11834