Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for "nested" mappings #645

Merged
merged 2 commits into from
Nov 7, 2022

Conversation

petardz
Copy link
Contributor

@petardz petardz commented Nov 4, 2022

Issue #, if available:

Description of changes:
Fixed bug where PUT _mapping against query index would fail in case of "nested" properties in index mappings

CheckList:
[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>
Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>
@petardz petardz requested a review from a team November 4, 2022 19:36
Comment on lines +149 to +159
client().admin().indices().putMapping(
PutMappingRequest(
index
).source("test_alias.field_a", "type=alias,path=message")
).get()
client().admin().indices().putMapping(
PutMappingRequest(
index
).source("test_alias2", "type=alias,path=test_field")
).get()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make a new test for nested situation?

@codecov-commenter
Copy link

codecov-commenter commented Nov 4, 2022

Codecov Report

Merging #645 (48a596b) into main (c6c12db) will decrease coverage by 0.31%.
The diff coverage is 81.25%.

@@             Coverage Diff              @@
##               main     #645      +/-   ##
============================================
- Coverage     76.37%   76.06%   -0.32%     
  Complexity      116      116              
============================================
  Files           124      124              
  Lines          6443     6497      +54     
  Branches        942      960      +18     
============================================
+ Hits           4921     4942      +21     
- Misses         1040     1066      +26     
- Partials        482      489       +7     
Impacted Files Coverage Δ
...opensearch/alerting/util/DocLevelMonitorQueries.kt 80.00% <81.25%> (+0.45%) ⬆️
.../alerting/transport/TransportIndexMonitorAction.kt 64.65% <0.00%> (-5.70%) ⬇️
...destinationmigration/DestinationConversionUtils.kt 71.11% <0.00%> (-1.12%) ⬇️
...nationmigration/DestinationMigrationUtilService.kt 81.45% <0.00%> (-0.81%) ⬇️
...rch/alerting/resthandler/RestIndexMonitorAction.kt 81.03% <0.00%> (+0.33%) ⬆️
.../kotlin/org/opensearch/alerting/util/IndexUtils.kt 72.91% <0.00%> (+2.08%) ⬆️
...alerting/transport/TransportSearchMonitorAction.kt 75.00% <0.00%> (+2.50%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@lezzago lezzago merged commit 42abf4d into opensearch-project:main Nov 7, 2022
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.4 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.4 2.4
# Navigate to the new working tree
cd .worktrees/backport-2.4
# Create a new branch
git switch --create backport/backport-645-to-2.4
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 42abf4d2247e5629694dfd922cefcce23d4bd207
# Push it to GitHub
git push --set-upstream origin backport/backport-645-to-2.4
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.4

Then, create a pull request where the base branch is 2.4 and the compare/head branch is backport/backport-645-to-2.4.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-645-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 42abf4d2247e5629694dfd922cefcce23d4bd207
# Push it to GitHub
git push --set-upstream origin backport/backport-645-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-645-to-2.x.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-645-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 42abf4d2247e5629694dfd922cefcce23d4bd207
# Push it to GitHub
git push --set-upstream origin backport/backport-645-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-645-to-2.x.

petardz added a commit to petardz/alerting that referenced this pull request Nov 8, 2022
* example

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* fixed updating mappings for queryIndex

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>
(cherry picked from commit 42abf4d)
eirsep pushed a commit to eirsep/alerting that referenced this pull request Nov 9, 2022
* example

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* fixed updating mappings for queryIndex

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>
petardz added a commit to petardz/alerting that referenced this pull request Jan 4, 2023
* example

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* fixed updating mappings for queryIndex

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>
eirsep pushed a commit that referenced this pull request Jan 4, 2023
* Added support for "nested" mappings (#645)

* example

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* fixed updating mappings for queryIndex

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* mappings traversal bug fix (#669)



Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>
lezzago pushed a commit to lezzago/alerting-opensearch that referenced this pull request Jun 8, 2023
* Added support for "nested" mappings (opensearch-project#645)

* example

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* fixed updating mappings for queryIndex

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* mappings traversal bug fix (opensearch-project#669)

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>
lezzago added a commit that referenced this pull request Jun 8, 2023
* Added exception check once the .opendistro-alerting-config index is b… (#650)

* Added exception check once the .opendistro-alerting-config index is being created

During .opendistro-alerting-config index creation, if ResourceAlreadyExists exception is being raised, the flow will check first if the index is in yellow state and then it will re-try to index monitor

Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

* Formating of the file fixed

Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

* refactored DeleteMonitor Action to be synchronious (#628) (#630)

* refactored DeleteMonitor Action to be synchronious

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>
Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com>

* [Backport 2.x] QueryIndex rollover when field mapping limit is reached (#729)

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* Mappings fix backport 2.x (#730)

* Added support for "nested" mappings (#645)

* example

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* fixed updating mappings for queryIndex

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* mappings traversal bug fix (#669)

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* Added unwrapping exception from core; added more debug logs (#728)

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* Add DataSources test for future backports

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* fix  percolator mapping error when having field name 'type' (#726)

Signed-off-by: Raj Chakravarthi <raj@icedome.ca>

* [BUG] ExecuteMonitor inserting metadata doc during dry run (#758)

* execute monitor bugfix

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* added IT

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* fixed created retval when skipIndex=true

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

---------

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* fix for ERROR alert state generation in doc-level monitors (#768)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* Adjusting max field index setting dynamically for query index (#776)

* added adjusting max field index setting dynamicly for query index

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* Multiple indices support in DocLevelMonitorInput (#784)

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* Doc transform 2.x backport (#853)

* conflict resovle - backport from main to 2.x

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* fixed module class names

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

---------

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* Update config index schema if needed at the start of each monitor execution (#849)

* Update config index schema if needed at the start of each monitor execution

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

* Mappings parsing fix (#851)

* fixed mappings parsing when field name named "properties" exists in mappings

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* message typo fix

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

---------

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* [Backport 2.x] Notification security fix (#861)

* Notification security fix (#852)

* added injecting whole user object in threadContext before calling notification APIs so that backend roles are available to notification plugin

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* compile fix

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* refactored user_info injection to use InjectSecurity

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* ktlint fix

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

---------

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>
(cherry picked from commit e0b7a5a)

* remove unneeded import

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>

---------

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>
Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com>
Co-authored-by: Ashish Agrawal <ashisagr@amazon.com>

* Fixed a bug that prevented alerts from being generated for doc level monitors that use wildcard characters in index names. (#894)

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* fixed security tests (#484) (#794)

* fixed security tests

Signed-off-by: Raj Chakravarthi <raj@icedome.ca>
(cherry picked from commit c51940f)

---------

Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>
Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>
Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>
Signed-off-by: Raj Chakravarthi <raj@icedome.ca>
Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: AWSHurneyt <hurneyt@amazon.com>
Co-authored-by: Stevan Buzejic <30922513+stevanbz@users.noreply.github.com>
Co-authored-by: Surya Sashank Nistala <snistala@amazon.com>
Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com>
Co-authored-by: RAJ CHAKRAVARTHI <49325334+raj-chak@users.noreply.github.com>
Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: AWSHurneyt <hurneyt@amazon.com>
Co-authored-by: RAJ CHAKRAVARTHI <raj@icedome.ca>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants