[Meta] Shard level Indexing Back-Pressure

[getsaurabh02]

Is your feature request related to a problem? Please describe.
Elastic search today provides few gating mechanism to protect a node when under a duress, via the concepts of queue rejections and circuit breakers. However, queue sizes are fixed, isolated and do not effectively represent the total work required to be done. Similarly, Circuit breakers acts as a last line of defence, are mostly too late to act upon, and do not offer fairness. Some of these gaps create availability issue with the cluster, when under duress due to hardware failures, node performance degradation or traffic bursts.

Indexing Pressure today tries to address this to some extent by rejecting indexing requests based on some hard-coded limits on nodes. However, there is a need for smarter rejection mechanism at shard level, when there are too many stuck/slow indexing requests, breaching key performance thresholds (such as throughput). This can prevent the cluster from running into cascading effects of failures.

Describe the solution you'd like
With shard level indexing pressure we want to improve the current Indexing Pressure framework which performs memory accounting at node level and rejects the requests. We aim to take a step further to have rejections based on the memory accounting at shard level along with other key performance factors like throughput and last successful requests. This can be called as ShardIndexingPressure.

Key features to be covered

• Granular tracking of indexing tasks performance, at every Shard level, for each Node role i.e. coordinator, primary and replica.
• Smarter rejections by discarding the requests intended only for problematic index or shard, while still allowing others to continue (fairness in rejection).
• Rejections thresholds governed by combination of configurable parameters (such as memory limits on node) and dynamic parameters (such as latency increase, throughput degradation).
• Node level and Shard level indexing pressure statistics exposed through stats api.
• Integration of Indexing pressure stats with Plugins for for metric visibility and auto-tuning in future.
• Control knobs to tune to the key performance thresholds which control rejections, to address any specific requirement or issues.
• Control knobs to run the feature in Shadow-Mode or Enforced-Mode. In shadow-mode only internal rejection breakdown metrics will be published while no actual rejections will be performed.

Additional context
Shard Indexing Pressure will be available in two phases ie Shadow & Enforced, via dynamic ES settings as below:
Enabled - To turn the shard indexing pressure feature turn on and off (default off initially).
Enforced - To run the feature in Shadow/Enforced mode if Enabled. In Shadow mode there will be no rejections but metrics will be published. In Enforced mode there will be actual rejection happening in the cluster.

Rejection Criteria - Three broad criteria for rejections as below:
Node Limit - Acts as the defence line if the utilisation by all the shards reaches the node limit assigned. i.e. 10% of heap. This effectively indicates a shard is unable to take in more traffic on the node.
Throughput degradation - This is to detect any hardware/software issue resulting into performance degradation. When node level occupancy is already breaching its soft-limit, and there exists a constant deterioration in the request turnaround at a shard level, additional requests are rejected until the system recovers.
Last Successful request - This is to address stuck requests or black hole scenarios. When node level occupancy is already breaching the soft-limit, and shard has multiple outstanding requests whole no request are turning around, then beyond a threshold additional requests will be actively discarded until the system recovers.

All the thresholds and key parameters will be available in for of dynamic ES settings for real time tuning.

Feature Branch: 478_indexBackPressure

[getsaurabh02]

Breaking the changes further into more granular PRs as below. This is to logically help the reviewers to visit the changes.

• Add Shard Indexing Pressure Settings (Add Shard Indexing Pressure Settings. (#478) #716)
• Add Shard Indexing Pressure Tracker (Add Shard Indexing Pressure Tracker. (#478) #717)
• Add IndexingPressure refactoring to Support Extension (Refactor IndexingPressure to allow extension. (#478) #718)
• Add ShardIndexingPressureStore to manage lifecycle of trackers (Add Shard Indexing Pressure Store (#478) #838).
• Add Memory Manager to control shard level memory utilisation. (Add Shard Indexing Pressure Memory Manager (#478) #945)
• Add framework level constructs to track shard indexing pressure. (Add ShardIndexingPressure framework level construct and Stats #1015)
• Add IndexingPressureService to perform orchestration. (Add Indexing Pressure Service which acts as orchestrator for IP #1084)
• Add plumbing logic for IndexingPressureService into NodeService (Add plumbing logic for IndexingPressureService in Transport Actions. #1113)
• Add shard indexing pressure metric/stats via rest end point (Add shard indexing pressure metric/stats via rest end point. #1171)
• Add shard indexing pressure IT (Add shard indexing pressure integration tests. #1198)

Below are the initial PRs for IndexingPressure changes, which are now being broken and merged into feature branch. Keeping here for discussion traceability.

• Add framework level constructs to track shard indexing pressure. (Add framework level constructs to track shard indexing pressure. #525)
• Add plumbing logic to invoke shard indexing pressure during write operation. (Add framework level constructs to track shard indexing pressure. #525)
• Add shard indexing pressure metric/stats via rest end point (Add shard indexing pressure metric/stats via rest end point #498)
• Add shard indexing pressure IT ( Add shard indexing pressure IT #499)

Below are List of Items for Future Follow Ups on the backpressure changes:

• Doc update once we have one.
• Generic way to expose metrics to plugins.
• Revisit modelling ShardIndexingPressureTracker with more concrete info such as Roles. (Issue to be created for discussion) (Add Shard Indexing Pressure Tracker. (#478) #717)
• With the remodelling of ShardIndexingPressureTracker, there is an opportunity to break the core logic of ShardIndexingPressure for reusability.
• Evaluate and refactor shard store cleanup strategy.

[getsaurabh02]

Closing this issue as changes are already merged as part of #1336