Tracing for deep search path #12103

dzane17 · 2024-01-31T02:08:34Z

Description

Tracing instrumentation at deep search path

Added spans at:

Coordinator node request
Coordinator node search phases

Testing

% curl -X PUT "localhost:9200/test2?pretty" -H 'Content-Type: application/json' -d'                                                                    
{                                                  
  "settings": {
    "number_of_shards": 10
  }
}'

% curl -X POST "localhost:9200/_bulk?pretty" -H 'Content-Type: application/json' -d'
{ "index" : { "_index" : "test2", "_id" : "1" } }
{ "field1" : "value1" }
{ "index" : { "_index" : "test2", "_id" : "2" } }
{ "field2" : "value2" }
'

% curl -XGET 'localhost:9200/_search?pretty&phase_took' -H 'X-Opaque-Id: my-id' -H 'Content-Type: application/json' -d'{
 "query": { "match_all": {} }
}'

% curl -XGET 'localhost:9200/_search?pretty&search_type=dfs_query_then_fetch&phase_took' -H 'X-Opaque-Id: my-id' -H 'Content-Type: application/json' -d'{
 "query": { "match_all": {} }
}'

search_type=dfs_query_then_fetch

Span Details

1. Coordinator node request

{
   "traceId":"c6a3a7a39cc2e81bdf964fd7d81dbe67",
   "spanId":"def6dc54cfe41cf7",
   "parentSpanId":"101e8811a19ff208",
   "name":"transport indices:data/read/search",
   "kind":1,
   "startTimeUnixNano":"1707279254012638047",
   "endTimeUnixNano":"1707279254017864739",
   "attributes":[
      {
         "key":"task_id",
         "value":{
            "intValue":"3360"
         }
      },
      {
         "key":"action",
         "value":{
            "stringValue":"indices:data/read/search"
         }
      },
      {
         "key":"thread.name",
         "value":{
            "stringValue":"opensearch[88665a158598.ant.amazon.com][transport_worker][T#9]"
         }
      },
      {
         "key":"total_hits",
         "value":{
            "intValue":"2"
         }
      }
   ],
   "status":{
      
   }
}

2. Coordinator node phase

{
   "traceId":"e522537a055adc6fc59667b58347942c",
   "spanId":"c8ef2d7ad2584a6c",
   "parentSpanId":"5b42e305497d91f8",
   "name":"Query",
   "kind":2,
   "startTimeUnixNano":"1702333271223157629",
   "endTimeUnixNano":"1702333271225489595",
   "attributes":[
      {
         "key":"thread.name",
         "value":{
            "stringValue":"opensearch[88665a158598.ant.amazon.com][transport_worker][T#8]"
         }
      }
   ],
   "status":{
      
   }
}

Related Issues

Resolves #8554

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-01-31T02:28:40Z

Compatibility status:

Checks if related components are compatible with change 659c479

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2024-01-31T02:44:06Z

❌ Gradle check result for 82843fd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-31T03:41:53Z

❌ Gradle check result for 7896045: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

reta · 2024-03-11T22:06:31Z

@dzane17 could you please rebase / merge with main (and resolve the conflicts)?

Signed-off-by: David Zane <davizane@amazon.com>

… from the SearchRequestOperationsListener Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: David Zane <davizane@amazon.com>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

github-actions · 2024-03-11T23:12:07Z

❌ Gradle check result for 1e9e593: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-03-11T23:57:02Z

❌ Gradle check result for 659c479: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

reta · 2024-03-12T12:24:33Z

❌ Gradle check result for 659c479: FAILURE

#10558
#12197
#11155

github-actions · 2024-03-12T13:24:02Z

❕ Gradle check result for 659c479: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

reta · 2024-03-12T13:33:43Z

@Gaganjuneja mind please re-taking a look? Thank you!

Gaganjuneja · 2024-03-12T13:34:58Z

@Gaganjuneja mind please re-taking a look? Thank you!

Sure, by today evening.

opensearch-trigger-bot · 2024-03-13T17:23:10Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12103-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 77a119319c14b0c1fa45dc85a618bf2429576b1e
# Push it to GitHub
git push --set-upstream origin backport/backport-12103-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-12103-to-2.x.

reta · 2024-03-13T17:41:59Z

@dzane17 could you please backport to 2.x manually? thank you

* Tracing for deep search path Signed-off-by: David Zane <davizane@amazon.com> * Refactor search phase tracing instrumentation to detach span creation from the SearchRequestOperationsListener Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Close WrappedPhase span Signed-off-by: David Zane <davizane@amazon.com> * Add asserting listener class Signed-off-by: David Zane <davizane@amazon.com> * Cleanup AbstractSearchAsyncActionTests tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Cleanup SearchAsyncActionTests tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Cleanup CanMatchPreFilterSearchPhaseTests tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: David Zane <davizane@amazon.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> (cherry picked from commit 77a1193)

* Tracing for deep search path Signed-off-by: David Zane <davizane@amazon.com> * Refactor search phase tracing instrumentation to detach span creation from the SearchRequestOperationsListener Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Close WrappedPhase span Signed-off-by: David Zane <davizane@amazon.com> * Add asserting listener class Signed-off-by: David Zane <davizane@amazon.com> * Cleanup AbstractSearchAsyncActionTests tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Cleanup SearchAsyncActionTests tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Cleanup CanMatchPreFilterSearchPhaseTests tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: David Zane <davizane@amazon.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <andriy.redko@aiven.io>

* Tracing for deep search path Signed-off-by: David Zane <davizane@amazon.com> * Refactor search phase tracing instrumentation to detach span creation from the SearchRequestOperationsListener Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Close WrappedPhase span Signed-off-by: David Zane <davizane@amazon.com> * Add asserting listener class Signed-off-by: David Zane <davizane@amazon.com> * Cleanup AbstractSearchAsyncActionTests tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Cleanup SearchAsyncActionTests tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Cleanup CanMatchPreFilterSearchPhaseTests tests Signed-off-by: Andriy Redko <andriy.redko@aiven.io> --------- Signed-off-by: David Zane <davizane@amazon.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

dzane17 force-pushed the tracing-2 branch from 7896045 to 82843fd Compare January 31, 2024 02:16

dzane17 marked this pull request as ready for review January 31, 2024 06:14

dzane17 and others added 7 commits March 11, 2024 18:04

Tracing for deep search path

123920d

Signed-off-by: David Zane <davizane@amazon.com>

Refactor search phase tracing instrumentation to detach span creation…

38f9c3b

… from the SearchRequestOperationsListener Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Close WrappedPhase span

e52aacb

Signed-off-by: David Zane <davizane@amazon.com>

Add asserting listener class

b8d2731

Signed-off-by: David Zane <davizane@amazon.com>

Cleanup AbstractSearchAsyncActionTests tests

d1c819e

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Cleanup SearchAsyncActionTests tests

b1c0c44

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Cleanup CanMatchPreFilterSearchPhaseTests tests

659c479

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

dzane17 force-pushed the tracing-2 branch from 1e9e593 to 659c479 Compare March 11, 2024 23:05

reta self-requested a review March 12, 2024 13:33

reta approved these changes Mar 12, 2024

View reviewed changes

Gaganjuneja approved these changes Mar 13, 2024

View reviewed changes

reta merged commit 77a1193 into opensearch-project:main Mar 13, 2024
34 checks passed

opensearch-trigger-bot bot added the backport-failed label Mar 13, 2024

dzane17 deleted the tracing-2 branch March 13, 2024 17:33

dzane17 mentioned this pull request Mar 13, 2024

[Backport 2.x] Tracing for deep search path (#12103) #12646

Merged

8 tasks

BrewTestBot mentioned this pull request Apr 3, 2024

opensearch 2.13.0 Homebrew/homebrew-core#167852

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tracing for deep search path #12103

Tracing for deep search path #12103

dzane17 commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024

github-actions bot commented Jan 31, 2024

reta commented Mar 11, 2024

github-actions bot commented Mar 11, 2024

github-actions bot commented Mar 11, 2024

reta commented Mar 12, 2024

github-actions bot commented Mar 12, 2024

reta commented Mar 12, 2024

Gaganjuneja commented Mar 12, 2024

opensearch-trigger-bot bot commented Mar 13, 2024

reta commented Mar 13, 2024

Tracing for deep search path #12103

Tracing for deep search path #12103

Conversation

dzane17 commented Jan 31, 2024 • edited Loading

Description

Testing

search_type=dfs_query_then_fetch

Span Details

1. Coordinator node request

2. Coordinator node phase

Related Issues

Check List

github-actions bot commented Jan 31, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Jan 31, 2024

github-actions bot commented Jan 31, 2024

reta commented Mar 11, 2024

github-actions bot commented Mar 11, 2024

github-actions bot commented Mar 11, 2024

reta commented Mar 12, 2024

github-actions bot commented Mar 12, 2024

reta commented Mar 12, 2024

Gaganjuneja commented Mar 12, 2024

opensearch-trigger-bot bot commented Mar 13, 2024

reta commented Mar 13, 2024

dzane17 commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading