Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.9] Backport #1302, #1355, #1432, #1435, #1445, #1403, #1430, #1441 to 2.9 #1469

Merged
merged 12 commits into from
Mar 14, 2024

Conversation

@eirsep eirsep changed the title Backport 2.9 optimize doc-level monitor execution workflow for datastreams (#1302) [Backport 2.9] Backport #1302 and #1355 to 2.9 Mar 13, 2024
@eirsep eirsep changed the title [Backport 2.9] Backport #1302 and #1355 to 2.9 [Backport 2.9] Backport #1302, #1355, #1432 to 2.9 Mar 13, 2024
sbcd90 and others added 10 commits March 14, 2024 01:00
…arch-project#1302)

* optimize doc-level monitor execution for datastreams

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* add more tests to address comments

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* add integTest for multiple datastreams inside a single index pattern

* add integTest for multiple datastreams inside a single index pattern

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

---------

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
…arch-project#1355)

* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

* Addressing review comments

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <goyamegh@amazon.com>

---------

Signed-off-by: Megha Goyal <goyamegh@amazon.com>
…the object mapping [metadata.source_to_query_index_mapping] (opensearch-project#1432) (opensearch-project#1434)

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
… percolate queries (opensearch-project#1435)

* add jvm aware and max docs settings for batching docs for percolate queries

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix stats logging

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add queryfieldnames field in findings mapping

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Joanne Wang <jowg@amazon.com>
…vel monitor instead of entire _source for each doc (opensearch-project#1441)

* optimize to fetch only fields relevant to doc level queries in doc level monitor

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix test for settings check

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix ktlint

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
…c level monitor execution (opensearch-project#1445)

* optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix tests

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* optimize check indices and execute to query only write index of aliases and datastreams during monitor creation

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix test

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add javadoc

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add tests to verify seq_no calculation

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
@eirsep eirsep changed the title [Backport 2.9] Backport #1302, #1355, #1432 to 2.9 [Backport 2.9] Backport #1302, #1355, #1432, #1435, #1445, #1403, #1430, #1441 to 2.9 Mar 14, 2024
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Copy link

codecov bot commented Mar 14, 2024

Codecov Report

Attention: Patch coverage is 78.65330% with 149 lines in your changes are missing coverage. Please review.

Project coverage is 68.64%. Comparing base (0269dd1) to head (e74b166).
Report is 2 commits behind head on 2.9.

Files Patch % Lines
.../opensearch/alerting/DocumentLevelMonitorRunner.kt 82.89% 31 Missing and 21 partials ⚠️
...n/org/opensearch/alerting/core/lock/LockService.kt 58.73% 45 Missing and 7 partials ⚠️
.../org/opensearch/alerting/MonitorMetadataService.kt 52.63% 9 Missing and 9 partials ⚠️
...opensearch/alerting/util/DocLevelMonitorQueries.kt 77.14% 5 Missing and 3 partials ⚠️
...in/org/opensearch/alerting/MonitorRunnerService.kt 88.00% 4 Missing and 2 partials ⚠️
.../kotlin/org/opensearch/alerting/util/IndexUtils.kt 70.58% 1 Missing and 4 partials ⚠️
...lerting/transport/TransportDeleteWorkflowAction.kt 60.00% 2 Missing ⚠️
.../alerting/transport/TransportIndexMonitorAction.kt 66.66% 0 Missing and 2 partials ⚠️
...lin/org/opensearch/alerting/core/lock/LockModel.kt 96.55% 0 Missing and 2 partials ⚠️
...opensearch/alerting/model/IndexExecutionContext.kt 88.88% 1 Missing ⚠️
... and 1 more
Additional details and impacted files
@@             Coverage Diff              @@
##                2.9    #1469      +/-   ##
============================================
+ Coverage     68.35%   68.64%   +0.29%     
- Complexity      120      143      +23     
============================================
  Files           160      162       +2     
  Lines         10740    11268     +528     
  Branches       1628     1697      +69     
============================================
+ Hits           7341     7735     +394     
- Misses         2680     2774      +94     
- Partials        719      759      +40     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jowg-amazon
Copy link
Collaborator

Are we backporting #1139 to 2.9?

@eirsep eirsep merged commit a82c4f1 into opensearch-project:2.9 Mar 14, 2024
10 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants