[CM] Reload not reloading always

[ph]

I have been testing #10491 and I have saw cases where the reload of configuration did not happen correctly. At first, I thought It was because of an aggressive reload period or fixed for the leaks but this is not the case. I can reproduce the issue all the time with the 6.6 branch.

Steps:

1. Edit the api/configurations.go and make sure that equals always return false, this will force the manager to reload the on each fetch.

2. Use a simple log generator to create an infinite file.

LOG_FILE = "/tmp/super.log"
File.delete(LOG_FILE) if File.exist?(LOG_FILE)
File.open(LOG_FILE, "w+") do |f|
  loop do
    print "."
    f.write("hello world\n")
  end
end

3. Use the following configuration:

configok: true
configs:
- type: filebeat.inputs
  blocks:
  - raw:
      paths:
      - /tmp/super.log
- type: output
  blocks:
  - raw:
      elasticsearch:
        hosts:
        - 127.0.0.1
        password: changeme
        username: elastic
      output: elasticsearch

4. set a 30s or less for the management.period option.

5. start filebeat with the following command:

./filebeat -v -e -d "centralmgmt,elasticsearch,harvester"    

You should see Filebeat read the file and send events to ES on startup, it will periodically fetch and reload the configuration and then it will stop sending events.

[ph]

I see a few empty types, this should not be possible with the kibana refactor.

[ph]

I've looked at the output, I believe they are fine, I think we correctly handle the inputs, I have added trace in the code, we don't stop the input when we reload the input this mean we still have somehow an harvester linked to the file.

{"load":{"1":4.8252,"15":4.1782,"5":4.6909,"norm":{"1":0.6031,"15":0.5223,"5":0.5864}}}}}}
2019-02-04T17:02:10.401-0500    DEBUG   [input] input/input.go:152      Run input
2019-02-04T17:02:10.401-0500    DEBUG   [input] log/input.go:174        Start next scan
2019-02-04T17:02:10.401-0500    DEBUG   [input] log/input.go:404        Check file for harvesting: /tmp/super.log
2019-02-04T17:02:14.748-0500    DEBUG   [input] log/input.go:494        Update existing file for harvesting: /tmp/super.log, offset: 3865932
2019-02-04T17:02:14.748-0500    DEBUG   [input] log/input.go:546        Harvester for file is still running: /tmp/super.log
2019-02-04T17:02:14.748-0500    DEBUG   [input] log/input.go:195        input states cleaned up. Before: 1, After: 1, Pending: 0
2019-02-04T17:02:14.810-0500    DEBUG   [centralmgmt]   management/manager.go:181       Retrieving new configurations from Kibana
eeekeys? filebeat.inputs

[ph]

Ok after more investigation the above is correct, since the 'reload' method hash the configuration and since the configuration did not change for the inputs we do not stop the inputs, the input stays alive and it should block on the queue when we swap the output and we receive the signal to continue sending events to the queue.

[ph]

Now I will take a look more at the logic for swapping outputs.

[ycombinator]

This issue should be resolved by the fixes made in #17381. If it reappears in the context of Agent, let's open a new issue.