Improve index changes monitoring

[ruflin]

Better index changes monitoring, that reduces the number of requests to Elasticsearch (due to global checkout checks). There was a discussion with the elasticsearch team about implementing the sequence number monitoring as a part of the system index plugin.

[ruflin]

@aleksmaus Would be great if you could add a bit more details here.

[aleksmaus]

When the Fleet Server monitors for the new actions documents in the index, it is using the document sequence numbers as described in the proposal
https://docs.google.com/document/d/15sF6IZdH5KfbJmkDP97qIpmbNOUBWUGbBOEFS1ZigvQ/edit#heading=h.s3gr9natpeyf

The Elasticsearch team proposed to expose the system index plugin API that would encapsulate the logic of checking on global checkpoint value until it is fully advances (all the documents are fully indexed up to the given document sequence number). Something like it is done here for example:
https://github.com/elastic/elasticsearch/blob/95a7eed9aa35f47b228e402508709b5bd6703cf4/x-pack/plugin/ccr/src/main/java/org/elasticsearch/xpack/ccr/action/ShardChangesAction.java#L386

This allows Fleet Server to avoid having multiple calls to Elasticsearch, while waiting/checking on global checkpoint status. The Fleet Server would call this new API and Elasticsearch would return the results only when the global check point becomes consistent with the new document sequence numbers.

Tim Brooks volunteered to work on this on Elasticsearch side.

[ph]

@aleksmaus we can close this one, can you link the elasticsearch's PR?

[aleksmaus]

There were two changes of the API on Elasticsearch side, here are two corresponding PRs on Fleet Server side:
#200
#254

The "phase 1" is completed at this point, but there are "phase 2" and "phase 3" plans to evolve Elasticsearch API in the next releases.