Move HTTP calls to Logstash from New() to Fetch()

[ycombinator]

Resolves #15276.

This PR makes the logstash/node and logstash/node_stats metricsets resilient to Logstash's unavailability.

Before this PR, if Logstash was not already running when Metricbeat was started up with the logstash-xpack module enabled, Metricbeat would immediately exit with an error since Logstash was unreachable.

With this PR, Metricbeat will keep running and retrying to reach Logstash periodically. This also allows Logstash to go away temporarily (say, for an upgrade) and keeps Metricbeat running.

Testing this PR

1. Build Metricbeat with this PR.

   cd $GOPATH/src/github.com/elastic/beats/metricbeat
   mage build
   

2. Enable the logstash-xpack module.

   ./metricbeat modules enable logstash-xpack
   

3. Start up Metricbeat (without starting up Logstash).

   ./metricbeat -e
   

4. Note that there are errors in the Metricbeat log about it not being able to connect to Logstash's APIs. But also ensure that Metricbeat keeps running.

5. Start up Logstash.

6. Check the Metricbeat log again. Ensure that eventually (< 1 minute), the connection errors go away.

7. Ensure that a .monitoring-logstash-*-mb-* index for today's date exists. Ensure that it contains recent (timestamp within last 20 seconds) documents of type:logstash_stats. Ensure that it contains exactly one document per Logstash pipeline of type:logstash_state (a new document is created per pipeline, per Logstash node (re)start).

   POST .monitoring-logstash-*/_search
   {
     "size": 0,
     "aggs": {
       "by_type": {
         "terms": {
           "field": "type",
           "size": 10
         },
         "aggs": {
           "latest_timestamp": {
             "max": {
               "field": "timestamp"
             }
           }
         }
       }
     }
   }
   

8. Stop Logstash.

9. Repeat steps 4-7.

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[ph]

@ycombinator The two init function looks really close to me could it be possible to move them to the logstash package and use the same?

[ycombinator]

@ph I extracted much of the common code into the logstash package. Please take a look at let me know what you think. Thanks!

[ph]

LGTM

[ycombinator]

CI failures are unrelated. Merging.

[mtojek]

Manual testing passed (BC1).

Metricbeat survived taking down logstash. Noticed expected errors while restarting:

2020-01-21T11:47:15.603+0100	ERROR	[logstash.node]	node/node.go:79	could not fetch node pipelines: error making http request: Get http://localhost:9600/_node/pipelines?graph=true: dial tcp [::1]:9600: connect: connection refused
2020-01-21T11:47:15.608+0100	ERROR	[logstash.node_stats]	node_stats/node_stats.go:72	error making http request: Get http://localhost:9600/: dial tcp [::1]:9600: connect: connection refused

2020-01-21T11:47:35.606+0100	ERROR	[logstash.node]	node/node.go:79	could not fetch node pipelines: error making http request: Get http://localhost:9600/_node/pipelines?graph=true: EOF
2020-01-21T11:47:35.608+0100	ERROR	[logstash.node_stats]	node_stats/node_stats.go:72	error making http request: Get http://localhost:9600/: EOF
2020-01-21T11:47:45.607+0100	ERROR	[logstash.node]	node/node.go:79	could not fetch node pipelines: error making http request: Get http://localhost:9600/_node/pipelines?graph=true: EOF
2020-01-21T11:47:45.610+0100	ERROR	[logstash.node_stats]	node_stats/node_stats.go:72	error making http request: Get http://localhost:9600/: EOF