[Backport 2.3] Manual backports of several PRs

[lezzago]

Issue #, if available:

Description of changes:
This PR manually backports PRs: #484, #728, #729, #628, #650, #726, #768, #758, #776, #784, #853, #849, #851, #861, #894

CheckList:
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[lezzago]

Successful integ test run:

./gradlew clean && ./gradlew :alerting:integTest
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.4.2
  OS Info               : Linux 5.4.241-160.348.amzn2int.x86_64 (amd64)
  JDK Version           : 17 (Amazon Corretto JDK)
  JAVA_HOME             : /usr/lib/jvm/java-17-amazon-corretto.x86_64
  Random Testing Seed   : 674F1C782C9D6F0E
  In FIPS 140 mode      : false
=======================================

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1s
3 actionable tasks: 2 executed, 1 up-to-date
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.4.2
  OS Info               : Linux 5.4.241-160.348.amzn2int.x86_64 (amd64)
  JDK Version           : 17 (Amazon Corretto JDK)
  JAVA_HOME             : /usr/lib/jvm/java-17-amazon-corretto.x86_64
  Random Testing Seed   : 2B68CF3517D3B33E
  In FIPS 140 mode      : false
=======================================

> Task :alerting-core:compileKotlin
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/core/src/main/kotlin/org/opensearch/alerting/core/action/node/ScheduledJobsStatsTransportAction.kt: (11, 44): 'BaseNodeRequest' is deprecated. Deprecated in Java
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/core/src/main/kotlin/org/opensearch/alerting/core/action/node/ScheduledJobsStatsTransportAction.kt: (119, 39): 'BaseNodeRequest' is deprecated. Deprecated in Java

> Task :alerting:compileKotlin
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/BucketLevelMonitorRunner.kt: (36, 9): The corresponding parameter in the supertype 'MonitorRunner' is named 'dryRun'. This may cause problems when calling this function with named arguments.
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt: (69, 9): The corresponding parameter in the supertype 'MonitorRunner' is named 'dryRun'. This may cause problems when calling this function with named arguments.
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt: (103, 60): Unchecked cast: MutableMap<String, Any> to MutableMap<String, MutableMap<String, Any>>
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt: (441, 59): Unnecessary non-null assertion (!!) on a non-null receiver of type ClusterService
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt: (589, 57): Unchecked cast: Any to MutableMap<String, Any>
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/MonitorMetadataService.kt: (167, 76): Unchecked cast: Map<String, Any> to MutableMap<String, MutableMap<String, Any>>
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/QueryLevelMonitorRunner.kt: (28, 9): The corresponding parameter in the supertype 'MonitorRunner' is named 'dryRun'. This may cause problems when calling this function with named arguments.
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/alerts/AlertIndices.kt: (219, 42): 'getter for isLocalNodeElectedMaster: Boolean' is deprecated. Deprecated in Java
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/alerts/AlertIndices.kt: (227, 42): 'getter for isLocalNodeElectedMaster: Boolean' is deprecated. Deprecated in Java
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/model/DocumentLevelTriggerRunResult.kt: (34, 51): No cast needed
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/model/Finding.kt: (40, 40): Access to companion object through parenthesized class name is deprecated. Please, add explicit Companion qualifier.
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/model/Finding.kt: (90, 20): '@JvmOverloads' annotation has no effect for methods without default arguments
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/model/FindingWithDocs.kt: (28, 34): Access to companion object through parenthesized class name is deprecated. Please, add explicit Companion qualifier.
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/model/Monitor.kt: (113, 31): Access to companion object through parenthesized class name is deprecated. Please, add explicit Companion qualifier.
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/model/Monitor.kt: (114, 33): Access to companion object through parenthesized class name is deprecated. Please, add explicit Companion qualifier.
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/model/MonitorMetadata.kt: (40, 51): Unchecked cast: (Mutable)Map<String!, Any!>! to MutableMap<String, String>
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/model/MonitorMetadata.kt: (50, 48): Unchecked cast: MutableMap<String, String> to MutableMap<String, Any>
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/model/MonitorMetadata.kt: (60, 86): Unchecked cast: MutableMap<String, String> to MutableMap<String, Any>
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/model/MonitorMetadata.kt: (100, 94): Unchecked cast: (Mutable)Map<String!, Any!>! to MutableMap<String, String>
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/model/action/ActionExecutionScope.kt: (141, 17): Parameter 'sin' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/resthandler/RestIndexMonitorAction.kt: (87, 9): Non exhaustive 'when' statements on enum will be prohibited in 1.7, add 'CLUSTER_METRICS_MONITOR' branch or 'else' branch instead
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportDeleteMonitorAction.kt: (139, 43): Parameter 'monitor' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportDeleteMonitorAction.kt: (145, 17): Variable 'deleteResponse' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportDeleteMonitorAction.kt: (149, 17): Variable 'clusterState' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportDeleteMonitorAction.kt: (176, 21): Unsafe use of a nullable receiver of type TotalHits?
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportDeleteMonitorAction.kt: (187, 25): Variable 'response' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (126, 38): Unchecked cast: Any to MutableMap<String, Any>
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (133, 80): Unchecked cast: Any to MutableMap<String, Any>
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (137, 65): Unchecked cast: Any? to MutableMap<String, Any>
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (170, 16): Unnecessary safe call on a non-null receiver of type List<String>. This expression will have nullable type in future releases
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (176, 29): Unchecked cast: Any? to MutableMap<String, Any>
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (272, 37): Variable 'updateMappingResponse' initializer is redundant
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (300, 25): Name shadowed: unwrappedException
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (313, 21): Name shadowed: unwrappedException
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (343, 17): Variable 'updateSettingsResponse' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (356, 44): Parameter 'monitor' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/main/kotlin/org/opensearch/alerting/util/destinationmigration/DestinationMigrationCoordinator.kt: (77, 45): 'getter for isLocalNodeElectedMaster: Boolean' is deprecated. Deprecated in Java

> Task :alerting:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :alerting:compileTestKotlin
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (187, 13): Variable 'testDoc' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (522, 13): Variable 'searchResult' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (561, 13): Variable 'searchResult' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (614, 13): Variable 'searchResult' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (675, 13): Variable 'searchResult' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (316, 38): Variable 'executeMonitorResponse' initializer is redundant
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (613, 9): The value 'executeMonitor(updateMonitorResponse!!.monitor, monitorId, false)' assigned to 'var executeMonitorResponse: ExecuteMonitorResponse? defined in org.opensearch.alerting.MonitorDataSourcesIT.`test execute pre-existing monitor and update`' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (895, 27): Condition 'alerts != null' is always 'true'
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (913, 27): Condition 'alerts != null' is always 'true'
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (921, 27): Condition 'alerts != null' is always 'true'
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1014, 27): Condition 'alerts != null' is always 'true'
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1030, 27): Condition 'alerts != null' is always 'true'
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1038, 27): Condition 'alerts != null' is always 'true'
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1082, 36): Unnecessary safe call on a non-null receiver of type IndexMonitorResponse?. This expression will have nullable type in future releases
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1120, 9): The value 'monitorResponse!!.monitor' assigned to 'var monitor: Monitor defined in org.opensearch.alerting.MonitorDataSourcesIT.`test queryIndex gets increased max fields in mappings`' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1165, 31): Condition 'alerts != null' is always 'true'
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (102, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (162, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (183, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (226, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (270, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (318, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/model/DocLevelMonitorInputTests.kt: (66, 13): Variable 'inputString' is never used
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/resthandler/MonitorRestApiIT.kt: (868, 17): Name shadowed: alerts
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/transport/AlertingSingleNodeTestCase.kt: (71, 9): The expression is unused
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/transport/AlertingSingleNodeTestCase.kt: (82, 9): 'createIndex(String!, Settings!, String!, vararg String!): IndexService!' is deprecated. Deprecated in Java
w: /local/home/ashisagr/Documents/workspace/alerting-opensearch/alerting/src/test/kotlin/org/opensearch/alerting/transport/AlertingSingleNodeTestCase.kt: (116, 13): Variable 'foundAlias' is never used

> Task :alerting:integTest
OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=warn; support was removed in 17.0

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 4m 51s
16 actionable tasks: 16 executed