Added Point In Time Node Stats API ServiceLayer Changes #4030

ajaymovva · 2022-07-27T21:45:00Z

Signed-off-by: Ajay Kumar Movva movvaam@amazon.com

Description

This PR has service layer changes for the point in time stats in node stats api.

point_in_time_current : No of open pit contexts.
point_in_time_time_in_millis: Time pit contexts held open.
point_in_time_total: Completed pit contexts.

Sample Output.

Index Creation: localhost:9200/my-index-000001/_doc/.
Creation PIT with keep-alive as 1m,2m,3m.
Node Stats API output PIT's

when the pit requests are triggered.

"search": {
    "open_contexts": 3,
    "query_total": 0,
    "query_time_in_millis": 0,
    "query_current": 0,
    "fetch_total": 0,
    "fetch_time_in_millis": 0,
    "fetch_current": 0,
    "scroll_total": 0,
    "scroll_time_in_millis": 0,
    "scroll_current": 0,
    "point_in_time_total": 0,
    "point_in_time_time_in_millis": 0,
    "point_in_time_current": 3,
    "suggest_total": 0,
    "suggest_time_in_millis": 0,
    "suggest_current": 0
},

 when the 1m interval completed.

"search": {
    "open_contexts": 2,
    "query_total": 0,
    "query_time_in_millis": 0,
    "query_current": 0,
    "fetch_total": 0,
    "fetch_time_in_millis": 0,
    "fetch_current": 0,
    "scroll_total": 0,
    "scroll_time_in_millis": 0,
    "scroll_current": 0,
    "point_in_time_total": 1,
    "point_in_time_time_in_millis": 99516,
    "point_in_time_current": 2,
    "suggest_total": 0,
    "suggest_time_in_millis": 0,
    "suggest_current": 0
},

 when the 2m interval completed.

"search": {
    "open_contexts": 1,
    "query_total": 0,
    "query_time_in_millis": 0,
    "query_current": 0,
    "fetch_total": 0,
    "fetch_time_in_millis": 0,
    "fetch_current": 0,
    "scroll_total": 0,
    "scroll_time_in_millis": 0,
    "scroll_current": 0,
    "point_in_time_total": 2,
    "point_in_time_time_in_millis": 255747,
    "point_in_time_current": 1,
    "suggest_total": 0,
    "suggest_time_in_millis": 0,
    "suggest_current": 0
},

 when the 3m interval completed.

"search": {
    "open_contexts": 0,
    "query_total": 0,
    "query_time_in_millis": 0,
    "query_current": 0,
    "fetch_total": 0,
    "fetch_time_in_millis": 0,
    "fetch_current": 0,
    "scroll_total": 0,
    "scroll_time_in_millis": 0,
    "scroll_current": 0,
    "point_in_time_total": 3,
    "point_in_time_time_in_millis": 469004,
    "point_in_time_current": 0,
    "suggest_total": 0,
    "suggest_time_in_millis": 0,
    "suggest_current": 0
},

Issues Resolved

#1147

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

github-actions · 2022-07-27T22:10:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1054/
CommitID: f93ddc5

… into feature/pit_stats

github-actions · 2022-08-02T14:26:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1266/
CommitID: f46f7eb

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java

github-actions · 2022-08-03T08:21:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1340/
CommitID: 03dac6b73053558b1d2337461cf37f94dd8d2eee

github-actions · 2022-08-03T12:14:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1347/
CommitID: 00f6d86c9ef53eefb19287f92d2c360b4fb9ac0e

github-actions · 2022-08-03T12:19:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1349/
CommitID: 232c48e33f5af058828b844187dcb6a190cfcc2f

github-actions · 2022-08-03T12:29:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1354/
CommitID: f46f7eb

…eature/pit_stats Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

github-actions · 2022-08-03T12:46:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1355/
CommitID: 8a2afca

github-actions · 2022-08-03T13:03:09Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1357/
CommitID: 3eacc1d7c32b5b26965a1c44fd7b8df901fdee0e

github-actions · 2022-08-03T13:20:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1358/
CommitID: ec24fa413373296287dc13510d82ff938214fe4c

github-actions · 2022-08-03T13:27:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1359/
CommitID: 1d9a701f07b5d9862ac2bf32c7a3bbf507d83252

github-actions · 2022-08-03T13:33:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1360/
CommitID: 72f8abad83ca2bf4414cff3477ac96d3a61391f6

sachinpkale · 2022-08-04T05:07:33Z

Please add a sample output of node stats API in the PR description. I am not sure if I understand what each metric provides just by looking at the name. For example, from PR description, pit_total: Completed pit contexts, does it mean active PIT contexts are not counted under total? If so, why don't we name it as pit_completed?

ajaymovva · 2022-08-04T11:29:28Z

Please add a sample output of node stats API in the PR description. I am not sure if I understand what each metric provides just by looking at the name. For example, from PR description, pit_total: Completed pit contexts, does it mean active PIT contexts are not counted under total? If so, why don't we name it as pit_completed?

Added the Sample Output in the PR Description.
I agree that some of the stats can be modified into better naming. The reason why we used this format is to make sure we are inline with the other search related stats. All the other search stats are using the same format to emit the node stats response.

Reference: 

"search": {
    "open_contexts": 2,
    "query_total": 0,
    "query_time_in_millis": 0,
    "query_current": 0,
    "fetch_total": 0,
    "fetch_time_in_millis": 0,
    "fetch_current": 0,
    "scroll_total": 0,
    "scroll_time_in_millis": 0,
    "scroll_current": 0,
    "point_in_time_total": 1,
    "point_in_time_time_in_millis": 99516,
    "point_in_time_current": 2,
    "suggest_total": 0,
    "suggest_time_in_millis": 0,
    "suggest_current": 0
},

github-actions · 2022-08-04T11:34:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1440/
CommitID: bf2dfdc3ba721388a096132dad6ca36651c09bc7

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

github-actions · 2022-08-04T13:04:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/1441/
CommitID: 82aa086

reta · 2022-08-04T15:05:45Z

@ajaymovva could you please:

update api specs [1]
create a followup documentation issue [2] to document the payload changes

[1] https://github.com/opensearch-project/OpenSearch/blob/main/rest-api-spec/src/main/resources/rest-api-spec/test/cat.shards/10_basic.yml#L11
[2] https://github.com/opensearch-project/documentation-website/

ajaymovva · 2022-08-04T15:42:43Z

@ajaymovva could you please:

update api specs [1]

create a followup documentation issue [2] to document the payload changes

[1] https://github.com/opensearch-project/OpenSearch/blob/main/rest-api-spec/src/main/resources/rest-api-spec/test/cat.shards/10_basic.yml#L11 [2] https://github.com/opensearch-project/documentation-website/

This PR is only for the service layer changes related to node stats.

we will have the follow up rest layer PR which will have the change of update api specs.
we already have the documentation issue created for the pit changes. #674

sachinpkale · 2022-08-05T03:07:22Z

We use squash and merge policy while merging PRs. That means PR title becomes commit message for the merged PR. Please use imperative mood while writing PR title.

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java

github-actions · 2022-08-08T09:34:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/1557/
CommitID: e12ee9e

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

github-actions · 2022-08-08T10:36:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/1559/
CommitID: 7879a6b

codecov-commenter · 2022-08-08T10:39:08Z

Codecov Report

Merging #4030 (7879a6b) into main (c665e7c) will decrease coverage by 0.20%.
The diff coverage is 91.42%.

@@             Coverage Diff              @@
##               main    #4030      +/-   ##
============================================
- Coverage     70.62%   70.41%   -0.21%     
+ Complexity    57073    56967     -106     
============================================
  Files          4603     4603              
  Lines        274500   274535      +35     
  Branches      40206    40209       +3     
============================================
- Hits         193856   193319     -537     
- Misses        64393    64949     +556     
- Partials      16251    16267      +16

Impacted Files	Coverage Δ
...pensearch/index/search/stats/ShardSearchStats.java	`60.55% <90.90%> (+3.40%)`	⬆️
...org/opensearch/index/search/stats/SearchStats.java	`84.74% <91.66%> (+1.73%)`	⬆️
...java/org/opensearch/client/indices/DataStream.java	`0.00% <0.00%> (-76.09%)`	⬇️
.../opensearch/client/indices/CloseIndexResponse.java	`17.50% <0.00%> (-60.00%)`	⬇️
.../action/admin/indices/flush/ShardFlushRequest.java	`50.00% <0.00%> (-50.00%)`	⬇️
...cluster/coordination/PendingClusterStateStats.java	`20.00% <0.00%> (-48.00%)`	⬇️
...search/aggregations/pipeline/HoltWintersModel.java	`21.47% <0.00%> (-44.97%)`	⬇️
...regations/metrics/AbstractHyperLogLogPlusPlus.java	`51.72% <0.00%> (-44.83%)`	⬇️
.../java/org/opensearch/search/dfs/AggregatedDfs.java	`54.83% <0.00%> (-41.94%)`	⬇️
...pensearch/action/ingest/DeletePipelineRequest.java	`31.25% <0.00%> (-37.50%)`	⬇️
... and 475 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

…roject#4030) * Adds Node Stats api changes for the Point in time Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

Added Node Stats api changes for the Point in time

f93ddc5

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

ajaymovva marked this pull request as ready for review August 2, 2022 13:30

ajaymovva requested review from a team and reta as code owners August 2, 2022 13:30

Merge branch 'main' of https://github.com/opensearch-project/OpenSearch…

f46f7eb

… into feature/pit_stats

reta requested changes Aug 2, 2022

View reviewed changes

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java Outdated Show resolved Hide resolved

dhruv16dhr mentioned this pull request Aug 3, 2022

[META] Point In Time Search Issue List #3959

Open

47 tasks

ajaymovva force-pushed the feature/pit_stats branch from 232c48e to f46f7eb Compare August 3, 2022 12:12

Merge branch 'main' of https://github.com/ajaymovva/OpenSearch into f…

8a2afca

…eature/pit_stats Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

ajaymovva changed the title ~~Added Node Stats api changes for the Point in time~~ Added ServiceLayer Node Stats changes for the Point in time Aug 3, 2022

ajaymovva changed the title ~~Added ServiceLayer Node Stats changes for the Point in time~~ Point In Time Node Stats API Service Layer Changes Aug 3, 2022

ajaymovva force-pushed the feature/pit_stats branch from ec24fa4 to 1d9a701 Compare August 3, 2022 13:02

ajaymovva force-pushed the feature/pit_stats branch from 1d9a701 to 72f8aba Compare August 3, 2022 13:06

bharath-techie approved these changes Aug 3, 2022

View reviewed changes

ajaymovva force-pushed the feature/pit_stats branch from 72f8aba to bf2dfdc Compare August 4, 2022 11:18

Removed testDeleteAllPitReaderContexts test

82aa086

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

ajaymovva force-pushed the feature/pit_stats branch from bf2dfdc to 82aa086 Compare August 4, 2022 12:37

pranikum approved these changes Aug 4, 2022

View reviewed changes

ajaymovva requested a review from reta August 4, 2022 14:31

reta approved these changes Aug 4, 2022

View reviewed changes

ajaymovva changed the title ~~Point In Time Node Stats API Service Layer Changes~~ Added Point In time Node Stats API ServiceLayer Changes Aug 8, 2022

ajaymovva changed the title ~~Added Point In time Node Stats API ServiceLayer Changes~~ Added Point In Time Node Stats API ServiceLayer Changes Aug 8, 2022

Bukhtawar reviewed Aug 8, 2022

View reviewed changes

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java Show resolved Hide resolved

Bukhtawar reviewed Aug 8, 2022

View reviewed changes

server/src/main/java/org/opensearch/index/search/stats/SearchStats.java Outdated Show resolved Hide resolved

Merge branch 'opensearch-project:main' into feature/pit_stats

e12ee9e

Changing naming from pit to point_in_time

7879a6b

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

Bukhtawar merged commit 6993ac9 into opensearch-project:main Aug 8, 2022

ajaymovva mentioned this pull request Aug 18, 2022

Added RestLayer Changes for PIT stats #4217

Merged

5 tasks

bharath-techie mentioned this pull request Sep 5, 2022

[Backport 2.x] [Point in time] Backport point in time changes #4406

Closed

6 tasks

bharath-techie mentioned this pull request Sep 28, 2022

[Backport 2.x] [Point in time] Backport point in time changes #4616

Merged

6 tasks

This was referenced Dec 28, 2022

[BUG] Fix flaky org.opensearch.search.PitMultiNodeTests.testCreatePitWhileNodeDropWithAllowPartialCreationFalse test #4259

Open

Fix flaky PitMultiNodeTests including testCreatePitWhileNodeDropWithAllowPartialCreationFalse #5655

Merged

jed326 mentioned this pull request Aug 29, 2023

[Concurrent Segment Search] Explore different metrics/stats which will be useful with concurrent segment search #7359

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Point In Time Node Stats API ServiceLayer Changes #4030

Added Point In Time Node Stats API ServiceLayer Changes #4030

ajaymovva commented Jul 27, 2022 •

edited

Loading

github-actions bot commented Jul 27, 2022

github-actions bot commented Aug 2, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

github-actions bot commented Aug 3, 2022

sachinpkale commented Aug 4, 2022

ajaymovva commented Aug 4, 2022 •

edited

Loading

github-actions bot commented Aug 4, 2022

github-actions bot commented Aug 4, 2022

reta commented Aug 4, 2022

ajaymovva commented Aug 4, 2022 •

edited

Loading

sachinpkale commented Aug 5, 2022

github-actions bot commented Aug 8, 2022

github-actions bot commented Aug 8, 2022

codecov-commenter commented Aug 8, 2022

Added Point In Time Node Stats API ServiceLayer Changes #4030

Added Point In Time Node Stats API ServiceLayer Changes #4030

Conversation

ajaymovva commented Jul 27, 2022 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Jul 27, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 2, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 3, 2022

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Aug 4, 2022

ajaymovva commented Aug 4, 2022 • edited Loading

github-actions bot commented Aug 4, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 4, 2022

Gradle Check (Jenkins) Run Completed with:

reta commented Aug 4, 2022

ajaymovva commented Aug 4, 2022 • edited Loading

sachinpkale commented Aug 5, 2022

github-actions bot commented Aug 8, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 8, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Aug 8, 2022

Codecov Report

ajaymovva commented Jul 27, 2022 •

edited

Loading

ajaymovva commented Aug 4, 2022 •

edited

Loading

ajaymovva commented Aug 4, 2022 •

edited

Loading