Action Validation framework and Explain API integration #441

jowg-amazon · 2022-08-02T21:36:11Z

Issue #, if available:

Description of changes:

CheckList:

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

...n/kotlin/org.opensearch.indexmanagement.spi/indexstatemanagement/model/ValidationMetaData.kt

spi/src/main/kotlin/org.opensearch.indexmanagement.spi/indexstatemanagement/Validate.kt

...kotlin/org.opensearch.indexmanagement.spi/indexstatemanagement/model/ManagedIndexMetaData.kt

spi/src/main/kotlin/org.opensearch.indexmanagement.spi/indexstatemanagement/Validate.kt

src/main/kotlin/org/opensearch/indexmanagement/IndexManagementPlugin.kt

Signed-off-by: Joanne Wang <jowg@amazon.com>

…situations (opensearch-project#419) Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> Signed-off-by: Joanne Wang <jowg@amazon.com>

Signed-off-by: Joanne Wang <jowg@amazon.com>

…integration tests Signed-off-by: Joanne Wang <jowg@amazon.com>

Signed-off-by: Joanne Wang <jowg@amazon.com>

…pensearch-project#460) * Change test page size to avoid indexTimeInMillis < 1 issue. Signed-off-by: Angie Zhang <langelzh@amazon.com> * Change test page size to avoid indexTimeInMillis < 1 issue. Signed-off-by: Angie Zhang <langelzh@amazon.com> Signed-off-by: Angie Zhang <langelzh@amazon.com>

* transform maxClauses fix Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * added bucket log to track processed buckets Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * various renames/changes Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * fixed detekt issues Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * added comments to test Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * removed debug logging Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * empty commit to trigger checks Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * reduced pageSize to 1 in few ITs to avoid flaky tests; fixed bug where pagesProcessed was calculated incorrectly Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * reverted pagesProcessed change; fixed few ITs Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

opensearch-project#485) * 483: Updated detekt plugin and snakeyaml dependency. Updated a code to reduce the number of issues after static analysis Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * 483: Updated snakeyaml version to use the latest Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

…earch-project#471) (opensearch-project#107) Signed-off-by: Xuesong Luo <lxuesong@amazon.com> Signed-off-by: Xuesong Luo <lxuesong@amazon.com>

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

…pensearch-project#503)

Signed-off-by: cwillum <cwmmoore@amazon.com> Signed-off-by: cwillum <cwmmoore@amazon.com>

opensearch-project#490) * 64: Added rounding when using aggreagate script for avg metric. Added unit tests for checking average aggregations against the target rollup index Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * 64: Rollup job renamed Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * 64: Removed unrelevant metrics for the avg calculation test Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

…ect#528) * Revert Disable detekt: 50ac1e9 Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Remove force choosing snakeyml 1.31 Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Force snakeyaml 1.32 Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> * Empty commit Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com> Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com>

…) (opensearch-project#517) * Update 2.3 release note Signed-off-by: Angie Zhang <langelzh@amazon.com> * Update 2.3 release note Signed-off-by: Angie Zhang <langelzh@amazon.com> * Update 2.3 release note Signed-off-by: Angie Zhang <langelzh@amazon.com> * Update 2.3 release note Signed-off-by: Angie Zhang <langelzh@amazon.com> * Update 2.3 release note Signed-off-by: Angie Zhang <langelzh@amazon.com> Signed-off-by: Angie Zhang <langelzh@amazon.com> (cherry picked from commit d9793ac) Signed-off-by: Angie Zhang <langelzh@amazon.com> Signed-off-by: Angie Zhang <langelzh@amazon.com> (cherry picked from commit 7217b5b) Co-authored-by: Angie Zhang <langelzh@amazon.com>

opensearch-project#516) * Add 2.2 release note Signed-off-by: Angie Zhang <langelzh@amazon.com> * Add 2.2 release note Signed-off-by: Angie Zhang <langelzh@amazon.com> Co-authored-by: Angie Zhang <langelzh@amazon.com> (cherry picked from commit 8eb5da6) Signed-off-by: Angie Zhang <langelzh@amazon.com> Signed-off-by: Angie Zhang <langelzh@amazon.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com>

* [207]: Added 5 min scheduled job for sweeping ISM plugin version in the case of version discrepancy Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * [207]: Created pluginVersionSweepCoordinator component responsible for scheduling the skip execution task. Annotated tests in order to prevent thread leak error during integrational tests Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * [207]: Increased retry period for background job that sets the skip flag up to 5 mins Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Empty-Commit Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Co-authored-by: Stevan Buzejic <buzejic.stevan@gmail.com>

* flaky transform test fix attempt Signed-off-by: Petar Dzepina <petar.dzepina@vroom.com> * accidental paste fix Signed-off-by: Petar Dzepina <petar.dzepina@vroom.com> Signed-off-by: Petar Dzepina <petar.dzepina@vroom.com> Co-authored-by: Petar Dzepina <petar.dzepina@vroom.com>

codecov-commenter · 2022-10-13T21:38:52Z

Codecov Report

Merging #441 (6f74e73) into error-prevention-dev (2f7d009) will decrease coverage by 0.01%.
The diff coverage is 68.85%.

@@                    Coverage Diff                     @@
##             error-prevention-dev     #441      +/-   ##
==========================================================
- Coverage                   75.83%   75.81%   -0.02%     
- Complexity                   2477     2537      +60     
==========================================================
  Files                         314      321       +7     
  Lines                       14456    14755     +299     
  Branches                     2236     2287      +51     
==========================================================
+ Hits                        10963    11187     +224     
- Misses                       2248     2281      +33     
- Partials                     1245     1287      +42

Impacted Files	Coverage Δ
...dexmanagement/common/model/notification/Channel.kt	`48.38% <0.00%> (ø)`
...nt/indexstatemanagement/ManagedIndexCoordinator.kt	`69.29% <ø> (+0.58%)`	⬆️
...ndexstatemanagement/migration/MigrationServices.kt	`78.26% <ø> (ø)`
...nt/indexstatemanagement/model/destination/Chime.kt	`77.77% <0.00%> (+22.22%)`	⬆️
...statemanagement/model/destination/CustomWebhook.kt	`95.71% <0.00%> (+28.57%)`	⬆️
...nt/indexstatemanagement/model/destination/Slack.kt	`55.55% <0.00%> (-22.23%)`	⬇️
...t/settings/LegacyOpenDistroManagedIndexSettings.kt	`98.56% <ø> (ø)`
...opensearch/indexmanagement/rollup/RollupIndexer.kt	`56.06% <0.00%> (ø)`
.../opensearch/indexmanagement/rollup/RollupRunner.kt	`59.29% <ø> (ø)`
...management/engine/states/deletion/DeletingState.kt	`82.66% <0.00%> (-1.34%)`	⬇️
... and 47 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

jowg-amazon requested a review from a team August 2, 2022 21:36

bowenlan-amzn reviewed Aug 3, 2022

View reviewed changes

jowg-amazon changed the title ~~Explain integration~~ Action Validation framework and Explain API integration Aug 3, 2022

bowenlan-amzn changed the base branch from main to error-prevention-dev August 16, 2022 21:22

jowg-amazon and others added 26 commits August 18, 2022 23:13

initial framework

8bae0e3

Signed-off-by: Joanne Wang <jowg@amazon.com>

Removed recursion from Explain Action to avoid stackoverflow in some …

60edc73

…situations (opensearch-project#419) Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> Signed-off-by: Joanne Wang <jowg@amazon.com>

enabled by default integrated

0060bf0

Signed-off-by: Joanne Wang <jowg@amazon.com>

cleaned up comments and logs, created unit test and updated previous …

2aa150a

…integration tests Signed-off-by: Joanne Wang <jowg@amazon.com>

added delete validation logic

8add902

Signed-off-by: Joanne Wang <jowg@amazon.com>

fixed rollover validation unit tests

9041646

Signed-off-by: Joanne Wang <jowg@amazon.com>

added validation info field to ManagedIndexMetaData

0cda31a

Signed-off-by: Joanne Wang <jowg@amazon.com>

removed step context as input

daa7cdb

Signed-off-by: Joanne Wang <jowg@amazon.com>

added validationmetadata class

acec69f

Signed-off-by: Joanne Wang <jowg@amazon.com>

restored old integration tests and changed validation service output

78eabc9

Signed-off-by: Joanne Wang <jowg@amazon.com>

before integrated validation meta data into managed index meta data

31d7d45

Signed-off-by: Joanne Wang <jowg@amazon.com>

integrated validation meta data

8515efd

Signed-off-by: Joanne Wang <jowg@amazon.com>

working version

3771eca

Signed-off-by: Joanne Wang <jowg@amazon.com>

added validation mapping

1492da9

Signed-off-by: Joanne Wang <jowg@amazon.com>

fixed integ tests

2905ddf

Signed-off-by: Joanne Wang <jowg@amazon.com>

renamed some values

aceaf52

Signed-off-by: Joanne Wang <jowg@amazon.com>

before removing from managed index meta data

c5cd789

Signed-off-by: Joanne Wang <jowg@amazon.com>

created validation result object in explain

e060642

Signed-off-by: Joanne Wang <jowg@amazon.com>

testing

28e8715

Signed-off-by: Joanne Wang <jowg@amazon.com>

run fails

5e5244c

Signed-off-by: Joanne Wang <jowg@amazon.com>

integration test for delete + added framework for force merge

e1ff73a

Signed-off-by: Joanne Wang <jowg@amazon.com>

removed step validation metadata and still testing explain results

ace4998

Signed-off-by: Joanne Wang <jowg@amazon.com>

before removing from managed index runner

2a1203e

Signed-off-by: Joanne Wang <jowg@amazon.com>

removed from managed index runner

b0fc77c

Signed-off-by: Joanne Wang <jowg@amazon.com>

clean up and tests

bfb0c39

Signed-off-by: Joanne Wang <jowg@amazon.com>

all validation tests pass

a3edbe9

Signed-off-by: Joanne Wang <jowg@amazon.com>

jowg-amazon added 4 commits August 18, 2022 23:13

removed validation result from all managed index meta data

06cd2fc

Signed-off-by: Joanne Wang <jowg@amazon.com>

restored old IT tests

40e5f8e

Signed-off-by: Joanne Wang <jowg@amazon.com>

fixed it tests, set explain validation to false

71a2b9f

Signed-off-by: Joanne Wang <jowg@amazon.com>

clean up

9169b5b

Signed-off-by: Joanne Wang <jowg@amazon.com>

jowg-amazon force-pushed the explainIntegration branch from 32b2f78 to 9169b5b Compare August 18, 2022 23:13

jowg-amazon and others added 16 commits August 18, 2022 16:16

Merge branch 'error-prevention-dev' into explainIntegration

880a1bc

Remove HOST_DENY_LIST usage as Notification plugin will own it (opens…

39be4e3

…earch-project#471) (opensearch-project#107) Signed-off-by: Xuesong Luo <lxuesong@amazon.com> Signed-off-by: Xuesong Luo <lxuesong@amazon.com>

Disable detekt because of the CVE (opensearch-project#497)

50ac1e9

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

Deprecate Master nonmenclature (opensearch-project#501)

a9d04ec

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

[AUTO] Increment version to 2.3.0-SNAPSHOT (opensearch-project#484) (o…

4275b9f

…pensearch-project#503)

fix#921-README-forum-link-index_mgmnt (opensearch-project#499)

79214a2

Signed-off-by: cwillum <cwmmoore@amazon.com> Signed-off-by: cwillum <cwmmoore@amazon.com>

Merge branch 'error_prevention_dev' into explainIntegration

6f74e73

bowenlan-amzn approved these changes Oct 13, 2022

View reviewed changes

Angie-Zhang approved these changes Oct 14, 2022

View reviewed changes

Angie-Zhang merged commit 35b80e0 into opensearch-project:error-prevention-dev Oct 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Action Validation framework and Explain API integration #441

Action Validation framework and Explain API integration #441

jowg-amazon commented Aug 2, 2022 •

edited by bowenlan-amzn

Loading

codecov-commenter commented Oct 13, 2022 •

edited

Loading

Action Validation framework and Explain API integration #441

Action Validation framework and Explain API integration #441

Conversation

jowg-amazon commented Aug 2, 2022 • edited by bowenlan-amzn Loading

codecov-commenter commented Oct 13, 2022 • edited Loading

Codecov Report

jowg-amazon commented Aug 2, 2022 •

edited by bowenlan-amzn

Loading

codecov-commenter commented Oct 13, 2022 •

edited

Loading