[DOC] Search backpressure cluster settings documentation

[PritLadani]

What do you want to do?

• Request a change to existing documentation
• Add new documentation
• Report a technical problem with the documentation
• Other

Tell us about your request. Provide a summary of the request and all versions that are affected.
As a part of Search backpressure cancellation at the coordinator level, we have introduced a new dynamic cluster settings and deprecated a few existing dynamic settings. Using the new settings, customers can tune the search backpressure thresholds.

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.
Issue link: opensearch-project/OpenSearch#5173
PR: opensearch-project/OpenSearch#5605
Current public doc: https://opensearch.org/docs/latest/opensearch/search-backpressure#search-backpressure-settings

[PritLadani]

Newly introduced settings:

Setting	Default	Description
search_backpressure.search_task.total_heap_percent_threshold	5%	The heap usage threshold (as a percentage) required for the sum of heap usages of all search tasks before cancellation is applied.
search_backpressure.search_task.cpu_time_millis_threshold	30000	Defines the CPU usage threshold (in millis) for an individual parent task before it is considered for cancellation.
search_backpressure.search_task.elapsed_time_millis_threshold	45000	Defines the elapsed time threshold (in millis) for an individual parent task before it is considered for cancellation.
search_backpressure.search_task.heap_percent_threshold	2%	Defines the heap usage threshold (in percentage) for an individual parent task before it is considered for cancellation.
search_backpressure.search_task.heap_variance	2.0	Defines the heap usage variance for an individual parent task before it is considered for cancellation. A task is considered for cancellation when taskHeapUsage is greater than or equal to heapUsageMovingAverage * variance.
search_backpressure.search_task.heap_moving_average_window_size	10	Defines the window size to calculate the moving average of heap usage of completed parent tasks
search_backpressure.search_task.cancellation_ratio	0.1	The maximum number of SearchTasks to cancel, as a percentage of successful SearchTask completions.
search_backpressure.search_task.cancellation_rate	0.003	The maximum number of SearchTasks to cancel per millisecond of elapsed time.
search_backpressure.search_task.cancellation_burst	5	The maximum number of SearchTasks to cancel in a single iteration of the observer thread.

In addition to the above settings, we have also deprecated a few settings as mentioned below:

Setting
search_backpressure.cancellation_ratio
search_backpressure.cancellation_rate
search_backpressure.cancellation_burst

We have also introduced replacement settings for the above settings:

Setting	Default	Description
search_backpressure.search_shard_task.cancellation_ratio	0.1	The maximum number of SearchShardTasks to cancel, as a percentage of successful SearchShardTasks completions.
search_backpressure.search_shard_task.cancellation_rate	0.003	The maximum number of SearchShardTasks to cancel per millisecond of elapsed time.
search_backpressure.search_shard_task.cancellation_burst	10	The maximum number of SearchShardTasks to cancel in a single iteration of the observer thread.

[PritLadani]

After adding the above settings, /_cluster/settings response looks like this:

{
    "acknowledged": true,
    "persistent": {
        "search_backpressure": {
            "mode": "monitor_only",
            "cancellation_burst": "10.0",
            "cancellation_ratio": "0.1",
            "cancellation_rate": "0.003",
            "search_task": {
                "elapsed_time_millis_threshold": "45000",
                "heap_variance": "2.0",
                "heap_percent_threshold": "0.02",
                "cancellation_burst": "5.0",
                "cpu_time_millis_threshold": "30000",
                "cancellation_ratio": "0.1",
                "cancellation_rate": "0.003",
                "total_heap_percent_threshold": "0.05",
                "heap_moving_average_window_size": "100"
            },
            "node_duress": {
                "cpu_threshold": "0.9",
                "heap_threshold": "0.7",
                "num_successive_breaches": "3"
            },
            "search_shard_task": {
                "elapsed_time_millis_threshold": "30000",
                "heap_variance": "2.0",
                "heap_percent_threshold": "0.005",
                "cancellation_burst": "10.0",
                "cpu_time_millis_threshold": "15000",
                "cancellation_ratio": "0.1",
                "cancellation_rate": "0.003",
                "total_heap_percent_threshold": "0.05",
                "heap_moving_average_window_size": "100"
            }
        }
    },
    "transient": {}
}

[kolchfa-aws]

No PR yet but on track

[kolchfa-aws]

@PritLadani @reta Do you see value in keeping the deprecated settings in 2.6 documentation along with the new ones? Or does it make sense to completely replace the deprecated ones with the new ones?

[reta]

@PritLadani @reta Do you see value in keeping the deprecated settings in 2.6 documentation along with the new ones? Or does it make sense to completely replace the deprecated ones with the new ones?

I would mention both (old + new) since some people may migrate from 2.5 to 2.6 and run into this deprecations.

[PritLadani]

We may also need other documentation aspects as below:

Measuring resource consumption

An observer thread periodically measures the resource usage of the node. If OpenSearch determines that the node is under duress, OpenSearch examines the resource usage of each search shard task and compares it against configurable thresholds. OpenSearch considers CPU usage, heap usage, and elapsed time and assigns each task a cancellation score that is then used to cancel the most resource-intensive tasks.

Replace it with:

An observer thread periodically measures the resource usage of the node. If OpenSearch determines that the node is under duress, OpenSearch examines the resource usage of each search task and search shard task and compares it against configurable thresholds. OpenSearch considers CPU usage, heap usage, and elapsed time and assigns each task a cancellation score that is then used to cancel the most resource-intensive tasks.

Search Backpressure Stats API

Hyperlink to the text nodes stats API operation seems to be wrong. Please update the link to: https://opensearch.org/docs/2.4/api-reference/nodes-apis/nodes-stats/

EXAMPLE RESPONSE

Update the response as below:

{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "runTask",
  "nodes": {
    "T7aqO6zaQX-lt8XBWBYLsA": {
      "timestamp": 1667409521070,
      "name": "runTask-0",
      "transport_address": "127.0.0.1:9300",
      "host": "127.0.0.1",
      "ip": "127.0.0.1:9300",
      "roles": [
         
      ],
      "attributes": {
        "testattr": "test",
        "shard_indexing_pressure_enabled": "true"
      },
      "search_backpressure": {
        "search_task": {
          "resource_tracker_stats": {
            "heap_usage_tracker": {
              "cancellation_count": 57,
              "current_max_bytes": 5739204,
              "current_avg_bytes": 962465,
              "rolling_avg_bytes": 4009239
            },
            "elapsed_time_tracker": {
              "cancellation_count": 97,
              "current_max_millis": 15902,
              "current_avg_millis": 9705
            },
            "cpu_usage_tracker": {
              "cancellation_count": 64,
              "current_max_millis": 8483,
              "current_avg_millis": 7843
            }
          },
          "cancellation_stats": {
            "cancellation_count": 102,
            "cancellation_limit_reached_count": 25
          }
        },
        "search_shard_task": {
          "resource_tracker_stats": {
            "heap_usage_tracker": {
              "cancellation_count": 34,
              "current_max_bytes": 1203272,
              "current_avg_bytes": 700267,
              "rolling_avg_bytes": 1156270
            },
            "cpu_usage_tracker": {
              "cancellation_count": 318,
              "current_max_millis": 731,
              "current_avg_millis": 303
            },
            "elapsed_time_tracker": {
              "cancellation_count": 310,
              "current_max_millis": 1305,
              "current_avg_millis": 649
            }
          },
          "cancellation_stats": {
            "cancellation_count": 318,
            "cancellation_limit_reached_count": 97
          }
        },
        "mode": "enforced"
      }
    }
  }
}

Response fields

Add below mentioned 3 raws in the table at second, third and fourth index respectively.

Field Name	Data type	Description
search_backpressure.search_task	Object	Statistics specific to the search task.
search_backpressure.search_task.resource_tracker_stats	Object	Statistics about the current search tasks.
search_backpressure.search_task.cancellation_stats	Object	Statistics about the search tasks canceled since the node last restarted.

Also,

• change the description for the raw with Field Name as search_backpressure.search_shard_task.resource_tracker_stats to Statistics about the current search shard tasks..
• change the description for the raw with Field Name as search_backpressure.search_shard_task.calcellation_stats to Statistics about the search shard tasks canceled since the node last restarted..

The final table should look like this:

Field Name	Data type	Description
search_backpressure	Object	Statistics about search backpressure.
search_backpressure.search_task	Object	Statistics specific to the search task.
search_backpressure.search_task.resource_tracker_stats	Object	Statistics about the current search tasks.
search_backpressure.search_task.cancellation_stats	Object	Statistics about the search tasks canceled since the node last restarted.
search_backpressure.search_shard_task	Object	Statistics specific to the search shard task.
search_backpressure.search_shard_task.resource_tracker_stats	Object	Statistics about the current search shard tasks.
search_backpressure.search_shard_task.cancellation_stats	Object	Statistics about the search shard tasks canceled since the node last restarted.
search_backpressure.mode	String	The mode for search backpressure.

• Add the same hyperlinks for search_task -> resource_tracker_stats and cancellation_stats from search_shard_task.

nit: There's a type in Response fields table. Update search_backpressure.search_shard_task.calcellation_stats to search_backpressure.search_shard_task.cancellation_stats.

[PritLadani]

One more edit:
In the Search backpressure settings table, we should remove the second raw, i.e. search_backpressure.interval_millis since it is not a dynamic setting.

[PritLadani]

Also, we need to update the wording which describes the cancellation_count. Currently it says The total number of tasks canceled since the node last restarted.. However, it should be something like The total number of tasks marked for cancellation since the node last restarted., since marking a task for cancellation does not guarantee task cancellation.
Same goes for below fields as well:
cpu_usage_tracker -> cancellation_count: The number of tasks marked for cancellation because of excessive CPU usage since the node last restarted.
heap_usage_tracker -> cancellation_count: The number of tasks marked for cancellation because of excessive heap usage since the node last restarted.
elapsed_time_tracker -> cancellation_count: The number of tasks marked for cancellation because of excessive elapsed time since the node last restarted.

[kolchfa-aws]

The cancellation_count wording change is applicable to the current documentation and is addressed in #3033