[Search Pipelines] Add stats for search pipelines

[msfroh]

Description

This adds statistics on executions and time spent on search pipeline operations, similar to the stats that are available for ingest pipelines.

Related Issues

Resolves #6723

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17519/
• CommitID: a94b09a
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17520/
• CommitID: 97f64b3
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17522/
• CommitID: 3f3a7bb
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17599/
• CommitID: 0d3c6db
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/17628/
• CommitID: a6a5784
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8053 (159e568) into main (1cf9c5c) will increase coverage by 0.03%.
The diff coverage is 87.93%.

@@             Coverage Diff              @@
##               main    #8053      +/-   ##
============================================
+ Coverage     70.89%   70.92%   +0.03%     
- Complexity    56639    56656      +17     
============================================
  Files          4722     4725       +3     
  Lines        267674   267987     +313     
  Branches      39232    39267      +35     
============================================
+ Hits         189757   190063     +306     
+ Misses        61937    61864      -73     
- Partials      15980    16060      +80     

Impacted Files	Coverage Δ
...min/cluster/stats/TransportClusterStatsAction.java	69.56% <ø> (ø)
...src/main/java/org/opensearch/node/NodeService.java	73.49% <0.00%> (-0.90%)	⬇️
...rch/action/admin/cluster/node/stats/NodeStats.java	50.54% <33.33%> (-0.90%)	⬇️
.../action/admin/cluster/stats/ClusterStatsNodes.java	55.00% <55.55%> (+5.83%)	⬆️
.../org/opensearch/common/metrics/OperationStats.java	78.78% <78.78%> (ø)
.../java/org/opensearch/search/pipeline/Pipeline.java	87.83% <85.96%> (-1.82%)	⬇️
...pensearch/search/pipeline/SearchPipelineStats.java	89.44% <89.44%> (ø)
...pensearch/search/pipeline/PipelineWithMetrics.java	90.90% <90.90%> (ø)
...main/java/org/opensearch/ingest/IngestService.java	83.37% <93.75%> (-1.16%)	⬇️
...on/admin/cluster/node/stats/NodesStatsRequest.java	93.33% <100.00%> (+0.09%)	⬆️
... and 8 more

... and 481 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure
      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermarkWithAutoReleaseEnabled

• URL: https://build.ci.opensearch.org/job/gradle-check/18225/
• CommitID: 159e568
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[msfroh]

@saratvemulapalli -- Addressed your suggestions.

Windows precommit failed, but that seems to be a "Windows build server" thing and not a "this PR" thing.

[saratvemulapalli]

@saratvemulapalli -- Addressed your suggestions.

Windows precommit failed, but that seems to be a "Windows build server" thing and not a "this PR" thing.

Thanks @msfroh. I've triggered a re-run of window precommit. I haven't seen failures just in Windows platform.
If the failure persists we can dig in.

[saratvemulapalli]

Thanks @msfroh for taking care of the feedback.
LGTM!

[msfroh]

Thanks, @saratvemulapalli!

I resolved the open conversations, so it looks like there are no immediate merge blockers.

[msfroh]

Oh, the "Some checks were not successful" note seems to be there because the Mac OS precommit was canceled when the Windows precommit failed.

If someone can retry the Mac OS precommit, I think that should turn green.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8053-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 46c9a211b6b9490f6a7ac9425e946986cd51bed2
# Push it to GitHub
git push --set-upstream origin backport/backport-8053-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8053-to-2.x.

[dblock]

@msfroh needs manual backport if you want this in 2.x

[mingshl]

I will help with the backport to 2.x

[lukas-vlcek]

Hi,

I wanted to friendly point out that IMO PRs like this should have had the breaking label.

I am maintaining external plugin (specifically the Prometheus exporter plugin) and as you can imagine renaming some of the methods to get operational stats about Ingest operations had impact (as the plugin needs to call them).

Specifically:

nodeIngestStats.getIngestCount()        ->  getCount()
nodeIngestStats.getIngestFailedCount()  ->  getFailedCount()
nodeIngestStats.getIngestCurrent()      ->  getCurrent()
nodeIngestStats.getIngestTimeInMillis() ->  getTotalTimeInMillis()

I totally get it that given current state of API isolation it is possible that any internal renaming is possibly a "breaking" change but hey ... it is. I do not know if it makes sense to establish a new rule about how the "breaking" label is applied but the fact is that change like this can cause compilation error somewhere else.

Lukáš

[msfroh]

I totally get it that given current state of API isolation it is possible that any internal renaming is possibly a "breaking" change but hey ... it is. I do not know if it makes sense to establish a new rule about how the "breaking" label is applied but the fact is that change like this can cause compilation error somewhere else.

Oh, shoot. I didn't expect that.

I'm not sure what the best option is there. I'll aim to add the "breaking" label if I rename an existing method in future, at least.

[lukas-vlcek]

@msfroh Do not feel sorry for this.
With 2.9.0 there was another breaking change that impacted me, see #7508 (comment)

I also do not know what is the solution here. We could introduce something like "internal_API_change" label which will not go to the change log but can be used on GitHub to filter tickets but it would have to be applied consistently and there is no way how to achieve this (except by increasing the work load).