Backport/backport 1645 to 2.x sig test #284
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add table write operator and builder Signed-off-by: Chen Dai <daichen@amazon.com> * Add UT for all new classes Signed-off-by: Chen Dai <daichen@amazon.com> * Rename child field Signed-off-by: Chen Dai <daichen@amazon.com> * Add columns field Signed-off-by: Chen Dai <daichen@amazon.com> * Update javadoc to prepare PR Signed-off-by: Chen Dai <daichen@amazon.com> Signed-off-by: Chen Dai <daichen@amazon.com>
…nsearch-project#1169) (opensearch-project#1173) Signed-off-by: Peng Huo <penghuo@gmail.com> (cherry picked from commit 5073215) Co-authored-by: Peng Huo <penghuo@gmail.com>
* Add between grammar and in-memory impl Signed-off-by: Chen Dai <daichen@amazon.com> * Add comparison test for between Signed-off-by: Chen Dai <daichen@amazon.com> * Add doctest for between Signed-off-by: Chen Dai <daichen@amazon.com> * Add not between support Signed-off-by: Chen Dai <daichen@amazon.com> * Fix doctest failure Signed-off-by: Chen Dai <daichen@amazon.com> * Refactor to rewrite to basic comparison expression Signed-off-by: Chen Dai <daichen@amazon.com> * Clean up unused code Signed-off-by: Chen Dai <daichen@amazon.com> * Prepare to publish PR Signed-off-by: Chen Dai <daichen@amazon.com> Signed-off-by: Chen Dai <daichen@amazon.com>
…rch-project#1188) * Fix precedence by reordering grammar rule Signed-off-by: Chen Dai <daichen@amazon.com> * Fix precedence in PPL Signed-off-by: Chen Dai <daichen@amazon.com> Signed-off-by: Chen Dai <daichen@amazon.com> (cherry picked from commit ab46fdd) Co-authored-by: Chen Dai <daichen@amazon.com>
…pensearch-project#1198) * Updated UTC TIME Functions for like now FunctionProperties. Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com> * Finished updates to tests. Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com> * Removed unused import. Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com> * Addressed PR comments. Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com> * Removed extra import. Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com> * removed `DateTimeFunction` used within class. Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com> Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com> (cherry picked from commit 94b6bec) Co-authored-by: MitchellGale-BitQuill <104795536+MitchellGale-BitQuill@users.noreply.github.com>
…nsearch-project#1208) * Unquote from subquery alias Signed-off-by: Chen Dai <daichen@amazon.com> * Add comparison test case Signed-off-by: Chen Dai <daichen@amazon.com> * Add more comparison test case Signed-off-by: Chen Dai <daichen@amazon.com> * Update doc to remove limitations Signed-off-by: Chen Dai <daichen@amazon.com> Signed-off-by: Chen Dai <daichen@amazon.com> (cherry picked from commit 91ef34d) Co-authored-by: Chen Dai <daichen@amazon.com>
…pensearch-project#1067) (opensearch-project#1199) Change relevance functions that query fields to throw a SemanticCheckException when a field is queried that does not exist. Signed-off-by: forestmvey <forestv@bitquilltech.com> (cherry picked from commit d03c176) Co-authored-by: Forest Vey <36905077+forestmvey@users.noreply.github.com>
…pensearch-project#1214) Signed-off-by: GabeFernandez310 <gabrielf@bitquilltech.com> Signed-off-by: GabeFernandez310 <gabrielf@bitquilltech.com> (cherry picked from commit 2c46797) Co-authored-by: GabeFernandez310 <gabrielf@bitquilltech.com>
…ensearch-project#1191) (opensearch-project#1212) * Allow score, type and scalar function name as identifier Signed-off-by: Chen Dai <daichen@amazon.com> * Revert score and ignore failed IT Signed-off-by: Chen Dai <daichen@amazon.com> * Add comparison test to address PR comment Signed-off-by: Chen Dai <daichen@amazon.com> Signed-off-by: Chen Dai <daichen@amazon.com> (cherry picked from commit 2f4924a) Co-authored-by: Chen Dai <daichen@amazon.com>
…pensearch-project#1220) Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> (cherry picked from commit 683d92f) Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>
) (opensearch-project#1234) * Add `TIMEDIFF` and `DATEDIFF` functions. Signed-off-by: Yury-Fridlyand <yuryf@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: Max Ksyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yuryf@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: Max Ksyunz <maxk@bitquilltech.com> (cherry picked from commit 438c44d) Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>
opensearch-project#1224) (opensearch-project#1233) Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: dblock <dblock@amazon.com> (cherry picked from commit f4ab469) Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
…oject#1209) Signed-off-by: Zelin Hao <zelinhao@amazon.com>
…project#1166) Added Tests And Implementation For Match_Query, Match_Phrase, and Multi_Match Functions Signed-off-by: GabeFernandez310 <gabrielf@bitquilltech.com> Signed-off-by: GabeFernandez310 <gabrielf@bitquilltech.com> Signed-off-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
…Hot-Fix Hot Fix For CI Build (cherry picked from commit aae57a0) Signed-off-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
…ckport-1166-to-2.x [Backport 2.x] Add Alternate Syntax For Match_Query And Other Functions
…ensearch-project#1246) Signed-off-by: vamsi-amazon <reddyvam@amazon.com> Signed-off-by: vamsi-amazon <reddyvam@amazon.com> (cherry picked from commit ee949cc) Co-authored-by: vamsi-amazon <reddyvam@amazon.com>
…nsearch-project#1231) (opensearch-project#1237) Added Testing And Implementation For Second_Of_Minute Function Signed-off-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Signed-off-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> (cherry picked from commit dce7d0e) Co-authored-by: GabeFernandez310 <gabrielf@bitquilltech.com>
… (opensearch-project#1252) * Add functions `ADDTIME` and `SUBTIME`. (#132) Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> (cherry picked from commit 7630f87) Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>
…project#1228) (opensearch-project#1239) Added Implementation And Testing For Day_Of_Week Function Signed-off-by: GabeFernandez310 <gabrielf@bitquilltech.com> Signed-off-by: GabeFernandez310 <gabrielf@bitquilltech.com> (cherry picked from commit bac9c37) Co-authored-by: GabeFernandez310 <gabrielf@bitquilltech.com>
…tion (opensearch-project#1253) * Add Minute_Of_Hour Function As An Alias Of Minute Function (#196) (opensearch-project#1230) Added Testing And Implementation For Minute_Of_Hour Function Signed-off-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Signed-off-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> (cherry picked from commit 61e2374) * Added Missing Imports Signed-off-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Signed-off-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
…nctions (opensearch-project#1205) (opensearch-project#1255) * Added long fix for CEIL, CEILING and FLOOR functions using LONG instead of INT for RETURN. Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>
…roject#1229) (opensearch-project#1250) * Resolve sub object field in search hit source Signed-off-by: Chen Dai <daichen@amazon.com> * Rename to unexpanded object Signed-off-by: Chen Dai <daichen@amazon.com> * Update IT with where condition Signed-off-by: Chen Dai <daichen@amazon.com> * Fix test index mapping Signed-off-by: Chen Dai <daichen@amazon.com> Signed-off-by: Chen Dai <daichen@amazon.com> (cherry picked from commit 151f4cc) Co-authored-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Joshua Li <joshuali925@gmail.com>
…ch-project#1223) (opensearch-project#1258) Added Support And Tests For Time Type in day_of_year, week_of_year, month_of_year Functions Signed-off-by: GabeFernandez310 <gabrielf@bitquilltech.com> (cherry picked from commit 6e72f18) Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
…ect#1264) update build add dependency in jdbc maven artifact add build phase for git clone of sql-cli Signed-off-by: YANGDB <yang.db.dev@gmail.com>
…ckport/backport-1227-to-2.x [Backport 2.x] Add Day_Of_Month Function As An Alias Of DayOfMonth
opensearch-project#1269) * Add Hour_Of_Day Function As An Alias Of Hour Added Tests And ImplementationFor Hour_Of_Day Function Signed-off-by: GabeFernandez310 <gabrielf@bitquilltech.com> * Fixed Checkstyle Signed-off-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Signed-off-by: GabeFernandez310 <gabrielf@bitquilltech.com> Signed-off-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> (cherry picked from commit 9e67e5b) Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com>
…ct#1640) (opensearch-project#1670) Signed-off-by: forestmvey <forestv@bitquilltech.com> Co-authored-by: forestmvey <forestv@bitquilltech.com>
…on (opensearch-project#1657) (opensearch-project#1672) Signed-off-by: forestmvey <forestv@bitquilltech.com> Co-authored-by: forestmvey <forestv@bitquilltech.com>
…earch-project#1677) * Create new anonymizer for new engine (#266) * Created anonymizer listener for anonymizing SQL queries through the new engine Signed-off-by: Matthew Wells <matthew.wells@improving.com> * Update for review comments Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * added missing file header, change public variable to private Signed-off-by: Matthew Wells <matthew.wells@improving.com> --------- Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: Matthew Wells <matthew.wells@improving.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> (cherry picked from commit 62120fd) Co-authored-by: Matthew Wells <matthew.wells@improving.com>
…rch-project#1682) * Support Alternate Datetime Formats (#268) * Add OpenSearchDateType as a datatype for matching with Date/Time OpenSearch types Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> --------- Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Signed-off-by: Guian Gumpac <guian.gumpac@improving.com> Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: MaxKsyunz <maxk@bitquilltech.com> (cherry picked from commit 986db39) Co-authored-by: Guian Gumpac <guian.gumpac@improving.com>
…<table>` queries (opensearch-project#1684) * Support for pagination in v2 engine of `SELECT * FROM <table>` queries (opensearch-project#1666) v2 SQL engine can now paginate simple queries. Pagination is initiated by setting fetch_size property in the request JSON. Pagination is implemented using the OpenSearch Scroll API. Please see pagination-v2.md for implementation details. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Max Ksyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> (cherry picked from commit 57ce303) * Fix test build failure. Somehow the import is required in 2.x but not 3.0 Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: Max Ksyunz <maxk@bitquilltech.com>
…pensearch-project#1689) * Allow backtick around fields in sort cmd Signed-off-by: Joshua Li <joshuali925@gmail.com> * Add test Signed-off-by: Joshua Li <joshuali925@gmail.com> * Update test setup Signed-off-by: Joshua Li <joshuali925@gmail.com> * revert ignoring sql-cli repo if exists Signed-off-by: Joshua Li <joshuali925@gmail.com> --------- Signed-off-by: Joshua Li <joshuali925@gmail.com> (cherry picked from commit 691012d) Co-authored-by: Joshua Li <joshuali925@gmail.com>
…pensearch-project#1729) (cherry picked from commit bd9b180)
Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com>
…pensearch-project#1742) Signed-off-by: Max Ksyunz <maxk@bitquilltech.com> (cherry picked from commit 29f99aa) Co-authored-by: Max Ksyunz <maxk@bitquilltech.com>
… (opensearch-project#1716) (opensearch-project#1734) * Fixed bug of byte/short values not handling divide/modulus arithmetic equations Signed-off-by: Matthew Wells <matthew.wells@improving.com> (cherry picked from commit 2c80631) Co-authored-by: Matthew Wells <matthew.wells@improving.com>
…lause and for functions and expressions in the query. (opensearch-project#1500) (opensearch-project#1741) * Add support for `WHERE` clause, column list in `SELECT` clause and for functions and expressions in the query. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix merge issue and address PR feedback by updating comments. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * More comments. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add extra check for unset `initialSearchRequest`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> (cherry picked from commit da386e5) Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>
…ROM`. (opensearch-project#1599) (opensearch-project#1745) * Support `ORDER BY` clauses in pagination and queries without `FROM`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix IT. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> (cherry picked from commit 94d5479) Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>
* Fix ML-commons missing dependency. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix `mockito` dependency. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Revert changes in `:opensearch` since it is not needed anymore. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> (cherry picked from commit f6e2a97) Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>
…s upon project failure in startOpenSearch (opensearch-project#1763) * fixed buildgradle merge conflicts Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com> * Fix CI (opensearch-project#1760) * Fix ML-commons missing dependency. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix `mockito` dependency. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Revert changes in `:opensearch` since it is not needed anymore. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>
…ct#1725) (opensearch-project#1770) * fixed round documentation Signed-off-by: Matthew Wells <matthew.wells@improving.com> (cherry picked from commit 34cad6e) Co-authored-by: Matthew Wells <matthew.wells@improving.com>
(cherry picked from commit c7dfdb3) Signed-off-by: forestmvey <forestv@bitquilltech.com> Co-authored-by: Forest Vey <36905077+forestmvey@users.noreply.github.com>
…ensearch-project#1751) Signed-off-by: Joshua Li <joshuali925@gmail.com> (cherry picked from commit 7525bb1) Co-authored-by: Joshua Li <joshuali925@gmail.com>
* Update SQL plugin for core refactor (opensearch-project#1571) Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix plugin compilation (opensearch-project#1580) * Changed gradle version and removed values iterator Signed-off-by: Guian Gumpac <guian.gumpac@improving.com> * Update a test to match new indexResponse.aliases() type. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Ran ./gradlew wrapper Signed-off-by: Guian Gumpac <guian.gumpac@improving.com> --------- Signed-off-by: Guian Gumpac <guian.gumpac@improving.com> Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Update sqlite-jdbc to 3.41.2.2 to address CVE-2023-32697 (opensearch-project#1667) * Update sqlite-jdbc to 3.41.2.2 to address CVE-2023-32697 Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Don't check column names on H2 results for correctness tests as described in opensearch-project#1667 (comment). Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address PR review comment. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Guian Gumpac <guian.gumpac@improving.com> Co-authored-by: MaxKsyunz <maxk@bitquilltech.com>
…ct#1737) (opensearch-project#1791) * Add support for Array and ExprValue Parsing With Inner Hits * Adding schema validation for IT test, and another UT for nested arrays. * Making handleAggregationResponse a private function. --------- (cherry picked from commit 9fbcf11) Signed-off-by: forestmvey <forestv@bitquilltech.com> Co-authored-by: Forest Vey <forestv@bitquilltech.com>
…t#1789) (opensearch-project#1794) * Add Support for Nested Function in Order By Clause (#280) * Adding order by clause support for nested function. Signed-off-by: forestmvey <forestv@bitquilltech.com> * Adding test coverage for nested in ORDER BY clause. Signed-off-by: forestmvey <forestv@bitquilltech.com> * Added nested function validation to NestedAnalyzer. Signed-off-by: forestmvey <forestv@bitquilltech.com> --------- Signed-off-by: forestmvey <forestv@bitquilltech.com> * Adding semantic check for missing arguments in function and unit test. Signed-off-by: forestmvey <forestv@bitquilltech.com> --------- Signed-off-by: forestmvey <forestv@bitquilltech.com> (cherry picked from commit 3302ec8) Co-authored-by: Forest Vey <forestv@bitquilltech.com>
* Fix CSV/RAW output header being application/json rather than plain/text (opensearch-project#1779) * Fix CI (opensearch-project#1760) * Fix ML-commons missing dependency. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix `mockito` dependency. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Revert changes in `:opensearch` since it is not needed anymore. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Matthew Wells <matthew.wells@improving.com> * Fix CSV/RAW outputting wrong format (#279) * Fixed bug where CSV/RAW outputs as JSON rather than plain text Signed-off-by: Matthew Wells <matthew.wells@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Matthew Wells <matthew.wells@improving.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> (cherry picked from commit 1ec696d) * updated tests Signed-off-by: Matthew Wells <matthew.wells@improving.com> * updated tests to return name Signed-off-by: Matthew Wells <matthew.wells@improving.com> * changed tests to return value Signed-off-by: Matthew Wells <matthew.wells@improving.com> * removed unneeded imports Signed-off-by: Matthew Wells <matthew.wells@improving.com> --------- Signed-off-by: Matthew Wells <matthew.wells@improving.com>
…ensearch-project#1802) * Keep up with refactoring in OpenSearch. * Updating code formatting. --------- (cherry picked from commit 3fc11a4) Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: Max Ksyunz <maxk@bitquilltech.com>
…roject#1645) Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> (cherry picked from commit 8e5d766) Signed-off-by: Max Ksyunz <max.ksyunz@improving.com>
andy-k-improving
pushed a commit
that referenced
this pull request
Nov 16, 2024
* Implement creation of ip2geo feature (#257) * Update gradle version to 7.6 (#265) Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com> * Implement creation of ip2geo feature * Implementation of ip2geo datasource creation * Implementation of ip2geo processor creation Signed-off-by: Heemin Kim <heemin@amazon.com> --------- Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com> Signed-off-by: Heemin Kim <heemin@amazon.com> Co-authored-by: Vijayan Balasubramanian <balasvij@amazon.com> * Added unit tests with some refactoring of codes (#271) * Add Unit tests * Set cache true for search query * Remove in memory cache implementation (Two way door decision) * Relying on search cache without custom cache * Renamed datasource state from FAILED to CREATE_FAILED * Renamed class name from *Helper to *Facade * Changed updateIntervalInDays to updateInterval * Changed value type of default update_interval from TimeValue to Long * Read setting value from cluster settings directly Signed-off-by: Heemin Kim <heemin@amazon.com> * Sync from main (#280) * Update gradle version to 7.6 (#265) Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com> * Exclude lombok generated code from jacoco coverage report (#268) Signed-off-by: Heemin Kim <heemin@amazon.com> * Make jacoco report to be generated faster in local (#267) Signed-off-by: Heemin Kim <heemin@amazon.com> * Update dependency org.json:json to v20230227 (#273) Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com> * Baseline owners and maintainers (#275) Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com> --------- Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com> Signed-off-by: Heemin Kim <heemin@amazon.com> Co-authored-by: Vijayan Balasubramanian <balasvij@amazon.com> Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com> * Add datasource name validation (#281) Signed-off-by: Heemin Kim <heemin@amazon.com> * Refactoring of code (#282) 1. Change variable name from datasourceName to name 2. Change variable name from id to name 3. Added helper methods in test code Signed-off-by: Heemin Kim <heemin@amazon.com> * Change field name from md5 to sha256 (#285) Signed-off-by: Heemin Kim <heemin@amazon.com> * Implement get datasource api (#279) Signed-off-by: Heemin Kim <heemin@amazon.com> * Update index option (#284) 1. Make geodata index as hidden 2. Make geodata index as read only allow delete after creation is done 3. Refresh datasource index immediately after update Signed-off-by: Heemin Kim <heemin@amazon.com> * Make some fields in manifest file as mandatory (#289) Signed-off-by: Heemin Kim <heemin@amazon.com> * Create datasource index explicitly (#283) Signed-off-by: Heemin Kim <heemin@amazon.com> * Add wrapper class of job scheduler lock service (#290) Signed-off-by: Heemin Kim <heemin@amazon.com> * Remove all unused client attributes (#293) Signed-off-by: Heemin Kim <heemin@amazon.com> * Update copyright header (#298) Signed-off-by: Heemin Kim <heemin@amazon.com> * Run system index handling code with stashed thread context (#297) Signed-off-by: Heemin Kim <heemin@amazon.com> * Reduce lock duration and renew the lock during update (#299) Signed-off-by: Heemin Kim <heemin@amazon.com> * Implements delete datasource API (#291) Signed-off-by: Heemin Kim <heemin@amazon.com> * Set User-Agent in http request (#300) Signed-off-by: Heemin Kim <heemin@amazon.com> * Implement datasource update API (#292) Signed-off-by: Heemin Kim <heemin@amazon.com> * Refactoring test code (#302) Make buildGeoJSONFeatureProcessorConfig method to be more general Signed-off-by: Heemin Kim <heemin@amazon.com> * Add ip2geo processor integ test for failure case (#303) Signed-off-by: Heemin Kim <heemin@amazon.com> * Bug fix and refactoring of code (#305) 1. Bugfix: Ingest metadata can be null if there is no processor created 2. Refactoring: Moved private method to another class for better testing support 3. Refactoring: Set some private static final variable as public so that unit test can use it 4. Refactoring: Changed string value to static variable Signed-off-by: Heemin Kim <heemin@amazon.com> * Add integration test for Ip2GeoProcessor (#306) Signed-off-by: Heemin Kim <heemin@amazon.com> * Add ConcurrentModificationException (#308) Signed-off-by: Heemin Kim <heemin@amazon.com> * Add integration test for UpdateDatasource API (#307) Signed-off-by: Heemin Kim <heemin@amazon.com> * Bug fix on lock management and few performance improvements (#310) * Release lock before response back to caller for update/delete API * Release lock in background task for creation API * Change index settings to improve indexing performance Signed-off-by: Heemin Kim <heemin@amazon.com> * Change index setting from read_only_allow_delete to write (#311) read_only_allow_delete does not block write to an index. The disk-based shard allocator may add and remove this block automatically. Therefore, use index.blocks.write instead. Signed-off-by: Heemin Kim <heemin@amazon.com> * Fix bug in get datasource API and improve memory usage (#313) Signed-off-by: Heemin Kim <heemin@amazon.com> * Change package for Strings.hasText (#314) (#317) Signed-off-by: Heemin Kim <heemin@amazon.com> * Remove jitter and move index setting from DatasourceFacade to DatasourceExtension (#319) Signed-off-by: Heemin Kim <heemin@amazon.com> * Do not index blank value and do not enrich null property (#320) Signed-off-by: Heemin Kim <heemin@amazon.com> * Move index setting keys to constants (#321) Signed-off-by: Heemin Kim <heemin@amazon.com> * Return null index name for expired data (#322) Return null index name for expired data so that it can be deleted by clean up process. Clean up process exclude current index from deleting. Signed-off-by: Heemin Kim <heemin@amazon.com> * Add new fields in datasource (#325) Signed-off-by: Heemin Kim <heemin@amazon.com> * Delete index once it is expired (#326) Signed-off-by: Heemin Kim <heemin@amazon.com> * Add restoring event listener (#328) In the listener, we trigger a geoip data update Signed-off-by: Heemin Kim <heemin@amazon.com> * Reverse forcemerge and refresh order (#331) Otherwise, opensearch does not clear old segment files Signed-off-by: Heemin Kim <heemin@amazon.com> * Removed parameter and settings (#332) * Removed first_only parameter * Removed max_concurrency and batch_size setting first_only parameter was added as current geoip processor has it. However, the parameter have no benefit for ip2geo processor as we don't do a sequantial search for array data but use multi search. max_concurrency and batch_size setting is removed as these are only reveal internal implementation and could be a future blocker to improve performance later. Signed-off-by: Heemin Kim <heemin@amazon.com> * Add a field in datasource for current index name (#333) Signed-off-by: Heemin Kim <heemin@amazon.com> * Delete GeoIP data indices after restoring complete (#334) We don't want to use restored GeoIP data indices. Therefore we delete the indices once restoring process complete. When GeoIP metadata index is restored, we create a new GeoIP data index instead. Signed-off-by: Heemin Kim <heemin@amazon.com> * Use bool query for array form of IPs (#335) Signed-off-by: Heemin Kim <heemin@amazon.com> * Run update/delete request in a new thread (#337) This is not to block transport thread Signed-off-by: Heemin Kim <heemin@amazon.com> * Remove IP2Geo processor validation (#336) Cannot query index to get data to validate IP2Geo processor. Will add validation when we decide to store some of data in cluster state metadata. Signed-off-by: Heemin Kim <heemin@amazon.com> * Acquire lock sychronously (#339) By acquiring lock asychronously, the remaining part of the code is being run by transport thread which does not allow blocking code. We want only single update happen in a node using single thread. However, it cannot be acheived if I acquire lock asynchronously and pass the listener. Signed-off-by: Heemin Kim <heemin@amazon.com> * Added a cache to store datasource metadata (#338) Signed-off-by: Heemin Kim <heemin@amazon.com> * Changed class name and package (#341) Signed-off-by: Heemin Kim <heemin@amazon.com> * Refactoring of code (#342) 1. Changed class name from Ip2GeoCache to Ip2GeoCachedDao 2. Moved the Ip2GeoCachedDao from cache to dao package Signed-off-by: Heemin Kim <heemin@amazon.com> * Add geo data cache (#340) Signed-off-by: Heemin Kim <heemin@amazon.com> * Add cache layer to reduce GeoIp data retrieval latency (opensearch-project#343) Signed-off-by: Heemin Kim <heemin@amazon.com> * Use _primary in query preference and few changes (opensearch-project#347) 1. Use _primary preference to get datasource metadata so that it can read the latest data. RefreshPolicy.IMMEDIATE won't refresh replica shards immediately according to #346 2. Update datasource metadata index mapping 3. Move batch size from static value to setting Signed-off-by: Heemin Kim <heemin@amazon.com> * Wait until GeoIP data to be replicated to all data nodes (opensearch-project#348) Signed-off-by: Heemin Kim <heemin@amazon.com> * Update packages according to a change in OpenSearch core (opensearch-project#354) * Update packages according to a change in OpenSearch core Signed-off-by: Heemin Kim <heemin@amazon.com> * Update packages according to a change in OpenSearch core (opensearch-project#353) Signed-off-by: Heemin Kim <heemin@amazon.com> --------- Signed-off-by: Heemin Kim <heemin@amazon.com> --------- Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com> Signed-off-by: Heemin Kim <heemin@amazon.com> Co-authored-by: Vijayan Balasubramanian <balasvij@amazon.com> Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.