Nats module improvements

[ChrsMark]

What this PR adds

This PR adds more metrics at connections and routes metricsets. Prior to this no metrics have been collected for each of the connections and for each of the routes. With this PR we collect metrics like in/out bytes etc for each the connections and for each of the routes. In order to have routes in a NATS server a more than 1 NATS servers should be presented creating a NATS cluster. For this docker-compose file of the module adds an extra NATS instance so as populate routes and their respective metrics.

Why it is important

It related to elastic/integrations#359. Next steps would be to enhance/improve dashboards.

How to test this manually

1. Start the nats cluster using docker-compose file of nats module
2. Enable nats metricbeat module connections and routes metricsets and point them to the monitoring port of the nats container (exported port of 8222 internal)
3. Make sure that connection and route details are being collected.

[elasticmachine]

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2247
Skipped	508
Total	2755

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: [Pull request #22445 updated, Started by user Chris Mark, Rebuilds build #21]

• Start Time: 2020-11-17T09:07:03.019+0000

• Duration: 82 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2253
Skipped	508
Total	2761

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2253
Skipped	508
Total	2761

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[jsoriano]

I think that a metricset that collects events per connection shouldn't be enabled by default.

With this change, we are modifying the behaviour of a default metricset from collecting a summary of metrics for all the connections, to collect one event per connection. I guess that a production nats server can have hundreds or thousands of connections at a given time, would this generate such number of events per fetch period?

I think that the summary, and the metrics per connection should be in different metricsets.

I see that we also collect the number of connections in the stats metricset (assuming that nats.stats.total_connections is the same thing).

Maybe a plan could be to:

• Leave the summarized metrics (the total count would be the only one at least by now) in the stats metricset.
• Deprecate the connections metricset in 7.x/7.11, and remove it in master/8.0, the only metric it collects is also collected by the stats metricset.
• Add a new, non-default connection metricset, that collects the metrics per connection. Other reason to move from connections to connection is that singular namespaces is usually preferred in metric names (e.g: system.process..., postgresql.statement...).

For the routes we could follow a similar plan, to deprecate routes and create a new non-default route metricset, but we would first need to add the count of routes to the stats metricset if possible.

[ChrsMark]

Added 2 new extra metricsets, disabled by default, to collect metrics per connection/route. After this we can deprecate connections and routes metricsets adding their sum under stats metricset.

[jsoriano]

Looks good, thanks for addressing all the comments. I think we could keep nats.server.time as deprecated instead of removing it, but I am also ok with removing it now, I don't think this field is used much.

[jsoriano]

We may want to keep this in existing modules to avoid breaking changes, though I don't think this field is going to be missed.

[jsoriano]

We may want to keep this value to avoid a smallish breaking change.