[Telemetry] [Monitoring] Only retry fetching usage once monitoring bulk upload is successful

[Bamieh]

The bulk uploader in monitoring attempts to bulk insert data into Elasticsearch every 10 seconds (defined by the flag xpack.monitoring.kibana.collection.interval).

To avoid performance issues, we have throttled fetching telemetry usage data to once every 24 hours in the bulk uploader when monitoring is enabled.

The current behavior is to keep fetching and trying to insert usage data until ES succeeds. Once it succeeds we start fetching usage every 24 hours.

When monitoring is not enabled, the bulk uploader will keep on retring since ES returns ignored: true (the index does not exist) rendering the operation as unsuccessful, hence fetching usage again.

This is happening on all 7.x and master. It was discovered when running a backport against 7.5 branch. (#54055)

To improve performance when monitoring is not enabled we can start fetching usage data once the bulk uploader gets a success on the bulk insert from ES.

The tiny downside to this approach is that we will not be getting usage data on the first successful insert after enabling monitoring. We will be getting this data on the second tick (in less that 20 seconds).

CC @aaronjcaldwell

[elasticmachine]

Pinging @elastic/pulse (Team:Pulse)

[TinaHeiligers]

@Bamieh I could reproduce the ES response having ignored: true in 7.5 and also in master.

[Bamieh]

@TinaHeiligers you are correct. I have updated the issue with more accurate description after i've debugged this further. I've submitted a PR to fix this so please have a look there as well 🙂 .

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)