Changing search shard failure logging to trace from debug

[sgup432]

Description

Currently we have logging level set to DEBUG for org.opensearch.action package like

logger.action.name = org.opensearch.action
logger.action.level = debug

Every time there is a failure on shard for example due to search queue size filling up, we log shard failure message along with complete exception stacktrace which is known to be expensive. So in scenarios, where there are a lot of queue rejections happening, this logging further degrades the cluster performance in terms of CPU, Heap, Disk I/O as mentioned here.

• As part of this change, we are changing shard failure messages to TRACE. If needed we can enable it via cluster settings on demand.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#7830

Alternate approach

• Another approach would have been to change opensearch.action log level to INFO but this would impact other indexing, cluster manager classes as well, so doesn't make sense.
• We could have also considered to change opensearch.action.search log level to INFO specifically but considering we have some useful non-harmful debug logs elsewhere, we don't want to do it for now.

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[sgup432]

@jainankitk @shwetathareja Can you take a quick look at this change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19227/
• CommitID: 189011d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermarkWithAutoReleaseEnabled

• URL: https://build.ci.opensearch.org/job/gradle-check/19311/
• CommitID: 2374312
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[sgup432]

@shwetathareja I would need help in merging this.

[sgup432]

@andrross @dblock @Bukhtawar Can anyone of you help me getting this reviewed or merged if looks okay?

[kotwanikunal]

@sgup432 There are a few unresolved comments. You have responded but please resolve these conversations (either you or the commenter) before I can merge it in.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testRelocateWhileContinuouslyIndexingAndWaitingForRefresh

• URL: https://build.ci.opensearch.org/job/gradle-check/20157/
• CommitID: 20bf733
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[sgup432]

@kotwanikunal I have resolved the comments.