Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run CI on JDK 8 #302

Merged
merged 125 commits into from
Mar 7, 2022
Merged

Run CI on JDK 8 #302

merged 125 commits into from
Mar 7, 2022

Conversation

martin-gaievski
Copy link
Member

@martin-gaievski martin-gaievski commented Mar 2, 2022 •
edited
Loading

Description

Adding JDK8 to CI matrix

Issues Resolved

100

Check List

  • Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@martin-gaievski martin-gaievski changed the title Adding JDK8 to CI matrix Run CI on JDK 8 Mar 2, 2022
@martin-gaievski
Copy link
Member Author

Build for JDK8 is failing both locally and in CI with following problem:

  • Where:
    Build file '/Users/gaievski/dev/opensearch/k-NN/build.gradle' line: 51

  • What went wrong:
    A problem occurred evaluating root project 'opensearch-knn'.

Failed to apply plugin 'opensearch.opensearchplugin'.
Could not create plugin of type 'PluginBuildPlugin'.
> Could not generate a decorated class for type PluginBuildPlugin.
> org/opensearch/gradle/plugin/PluginPropertiesExtension has been compiled by a more recent version of the Java Runtime (class file version 54.0), this version of the Java Runtime only recognizes class file versions up to 52.0

this is true for opensearch 1.2.0 and 1.30 SNAPSHOT versions.

@martin-gaievski
Copy link
Member Author

Seems this issue is blocked unless we'll be able to define dependency on opensearch artifact that has been built with JDK8. I've created a new issue for OpenSearch - opensearch-project/OpenSearch#2319

@dblock
Copy link
Member

dblock commented Mar 7, 2022 •
edited
Loading

@martin-gaievski This should work now, rebase and try again? Also fix DCO.

@vamshin vamshin added the v1.3.0 Issues and PRs related to version 1.3.0 label Mar 7, 2022
@codecov-commenter
Copy link

codecov-commenter commented Mar 7, 2022 •
edited
Loading

Codecov Report

Merging #302 (45003c2) into 1.x (fe22b85) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##                1.x     #302   +/-   ##
=========================================
  Coverage     83.38%   83.38%           
  Complexity      885      885           
=========================================
  Files           127      127           
  Lines          3834     3834           
  Branches        361      361           
=========================================
  Hits           3197     3197           
  Misses          475      475           
  Partials        162      162

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe22b85...45003c2. Read the comment docs.

VijayanB and others added 22 commits March 7, 2022 11:43
@VijayanB @martin-gaievski
Include codecov badge
b607c41 
Include code coverage as badge.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@jmazanec15 @martin-gaievski
Cherry-pick opensearch-project#54 to 1.x (opensearch-project#56)
659d9a8 
Signed-off-by: John Mazanec <jmazane@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Add release notes for 1.0.0.0
5fbc277 
Include release notes.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Update OpenSearch upstream to 1.0.0
9959367 
remove rc1 qualifier.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Update release note
0721399 
Including rc1 release notes and features that are merged after latest odfe release
to 1.0.0.0

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@jmazanec15 @martin-gaievski
Fix copyright in README and NOTICE (opensearch-project#77)
31ad22c 
Signed-off-by: John Mazanec <jmazane@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Add release notes for 1.0.0.0
eab8674 
Include release notes.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Update release note
bc11006 
Including rc1 release notes and features that are merged after latest odfe release
to 1.0.0.0

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@peterzhuamazon @martin-gaievski
Add Integtest.sh for OpenSearch integtest setups (opensearch-project#44)
11cfdb3 
* Add integtest automation

Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>

* correct the credential option param here

Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Fix snapshot build, build against OpenSearch 1.1
8cac072 
update ci to build against 1.x branch and use snapshot.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Update lib default version
3fe8dcc 
Use 1.1.0.0 as default value instead of local.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Fix invalid url
402a800 
Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Add release note for 1.1.0.0
5851913 
Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@dblock @martin-gaievski
Removed integtest.sh. (opensearch-project#97)
6f9b968 
Signed-off-by: dblock <dblock@dblock.org>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Update workflow
3d0018b 
main tracks 1.x and builds on 1.x from upstream.
Update version to 1.2.0-SNAPSHOT since next 1.x release is 1.2.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Use published daily snapshot dependencies
d99dea0 
Fetch dependencies from published locations in the following order
maven local, aws sonatype, central.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Add DCO check to workflow
c0a72c0 
Include workflow to check whether commits are signed.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VijayanB @martin-gaievski
Update branch pattern
de4670d 
Since feature branch will have "feature/" as prefix,
updated branch pattern to include feature branch as well.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@naveentatikonda @martin-gaievski
Increment version on main to 1.2.0.0 (opensearch-project#138)
4650cf3 
Signed-off-by: Naveen Tatikonda <navtat@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@jmazanec15 @martin-gaievski
Add faiss support in jni (opensearch-project#28)
164b167 
Signed-off-by: Jack Mazanec <jmazane1@nd.edu>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@jmazanec15 @martin-gaievski
Add index for storing template models (opensearch-project#34)
59e53b7 
Signed-off-by: Jack Mazanec <jmazane1@nd.edu>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@jmazanec15 @martin-gaievski
Add Google tests for unit testing jni (opensearch-project#36)
6a656c7 
Signed-off-by: Jack Mazanec <jmazane1@nd.edu>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
mend-for-github-com bot and others added 21 commits March 7, 2022 11:43
@mend-for-github-com @martin-gaievski
Add .whitesource configuration file (opensearch-project#249)
efde020 
Co-authored-by: whitesource-for-github-com[bot] <50673670+whitesource-for-github-com[bot]@users.noreply.github.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@jmazanec15 @martin-gaievski
Default to nmslib in scoring if engine isnt specified (opensearch-pro…
f3694da 
…ject#256)

Signed-off-by: John Mazanec <jmazane@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@martin-gaievski
Changing serialization for knn vector from single array object to col…
4aa0690 
…lection of floats (opensearch-project#253)

* Changing serialization for knn vector from single array object to collection of floats

rev2:
* Addressing PR comments:
- added getDefaultSerializer to Factory
- moved SerializationMode enum to a separate file
- added javadocs and comments
- adjust format, added missing endline characters
rev3:
* Addressing multiple review comments:
- replace Vector by KNNVector in class names and variables
- fixed method names in Serializer interface
- replace number of bytes in float from number to constant
rev4:
* Moving new classes under index.codec.util
rev5:
* Addressing multiple review comments:
- rework factory method getSerializerByStreamContent
- added test case for stream of unsupported content
- removed exceptions from Serializer interface method's signatures, changed it to unchecked runtime exception
- simplify license header in new classes

Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@martin-gaievski
Removing jcenter repo from build related gradle files (opensearch-pro…
a122e52 
…ject#261)

Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@martin-gaievski
Adding write permissions to backport action (opensearch-project#262)
4860429 
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@naveentatikonda @martin-gaievski
Add Recall Tests (opensearch-project#251)
91a7407 
* Add Recall Tests

Signed-off-by: Naveen Tatikonda <navtat@amazon.com>

* Calculate Recall using document ids and other minor changes

Signed-off-by: Naveen Tatikonda <navtat@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@martin-gaievski
Adding ExtensiblePlugin to KNNPlugin (opensearch-project#264)
70d0034 
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@jmazanec15 @martin-gaievski
Refactor benchmark dataset format and add big ann benchmark format (o…
8bdfea2 
…pensearch-project#265)

Signed-off-by: John Mazanec <jmazane@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@martin-gaievski
Adding gradle task for running integ tests in remote cluster (opensea…
a5e1795 
…rch-project#266)

* Adding gradle task for running integ tests in remote cluster

Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@jmazanec15 @martin-gaievski
Set default space type to L2 to support bwc (opensearch-project#267)
11511da 
This change adds L2 space type fall back during query.
This prevents backwards compatibility issues for clusters
upgrading from ES 7.1 and 7.4. In those versions, space
type was not introduced as a field attribute yet.

Signed-off-by: John Mazanec <jmazane@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@jmazanec15 @martin-gaievski
Change benchmark ingest took metric to total time (opensearch-project…
8d7c814 
…#268)

Measuring total took time will give a better picture of how long it
will take to make a particular set of documents searchable.

Signed-off-by: John Mazanec <jmazane@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@jmazanec15 @martin-gaievski
Make doc and query count configurable in benchmark (opensearch-projec…
2b85d0a 
…t#270)

Makes the document and query count configurable in the benchmarking
tool. With this functionality, users can now specify to only index or
search a subset of the vectors in the data set. This is useful for
indices that require training that may only need a subset of the data set
for training.

Signed-off-by: John Mazanec <jmazane@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@martin-gaievski
Adding JDK 11 to CI and docs (opensearch-project#271)
0ebeb12 
Adding JDK 11 to CI matrix of java versions, replacing as minimum prerequisite in Dev readme guide. Main reason for change - JDK 11 is the latest supported LTS.

Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@jmazanec15 @martin-gaievski
[Benchmark] Remove ingest results collection (opensearch-project#272)
a849b50 
Removes collecting ingest results from the benchmarking tool. On big
data sets, this will prevent the process from going out of memory.

Signed-off-by: John Mazanec <jmazane@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@github-actions @martin-gaievski
[BUG FIX] Add space type default and ef search parameter in warmup (o…
f6430c2 
…pensearch-project#279)

Sets the default value of space type to L2 in KNNIndexShard.
KNNIndexShard is used during warmup to load segments into memory. For
indices created in ES 7.1 and 7.4, they will not have this value set
because the only space we supported was l2. So, we need to hardcode the
defaults here.

For nmslib, the ef_search parameter is configurable at load time. So, it
needs to be passed as a parameter in both the search load phase as well
as warmup. This commit adds it to the warmup phase and abstracts load
parameters to a helper function so that it can be consistent for both
search and warmup.

Signed-off-by: John Mazanec <jmazane@amazon.com>
(cherry picked from commit 2fb2ad1)
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@github-actions @martin-gaievski
[FLAKY TEST] Fix codec test causing CI to fail (opensearch-project#277)…
e39b978 
… (opensearch-project#286)

Closes resource monitor at the end of the Codec tests so that there are no
unparked threads.

Signed-off-by: John Mazanec <jmazane@amazon.com>
(cherry picked from commit 18e6e35)
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@github-actions @martin-gaievski
Add CI to run every night (opensearch-project#284)
7883c1d 
Github doesn't support non-standard like @daily, we have to use
standard cron syntax.

Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
(cherry picked from commit 9c64451)
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@VachaShah @martin-gaievski
Adding auto delete workflow for backport branches (opensearch-project…
b9f0c51 
…#289) (opensearch-project#291)

Signed-off-by: Vacha Shah <vachshah@amazon.com>
(cherry picked from commit cb1a7d8)
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@opensearch-trigger-bot @martin-gaievski
Updates Guava versions to address CVE (opensearch-project#292) (opens…
045c8c5 
…earch-project#295)

Updates direct Guava dependency to 30.0. Also, updates check style to
9.3 to inherent Guava fix. Remove deprecated features from check style
  xml config file.

Signed-off-by: John Mazanec <jmazane@amazon.com>
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@opensearch-trigger-bot @martin-gaievski
[Backport 1.x] [CODE STYLE] Switch checkstyle to spotless (opensearch…
ca1c17d 
…-project#298)

Switches checkstyle functionality to spotless. Sets ratchetFrom to
origin/1.x. This will mean it will only fail on files recently changed.
Pull formatting from OpenSearch.

Signed-off-by: John Mazanec <jmazane@amazon.com>
(cherry picked from commit 0db9b2e)
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@martin-gaievski
Adding JDK8 support for CI matrix
0535672 
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
@martin-gaievski martin-gaievski marked this pull request as ready for review March 7, 2022 20:06
@martin-gaievski martin-gaievski requested a review from a team March 7, 2022 20:06
@martin-gaievski martin-gaievski merged commit 60766a4 into opensearch-project:1.x Mar 7, 2022
@martin-gaievski martin-gaievski mentioned this pull request Mar 11, 2022
Merged
1 task
martin-gaievski added a commit to martin-gaievski/k-NN that referenced this pull request Mar 31, 2022
@martin-gaievski
Run CI on JDK 8 (opensearch-project#302)
61c03eb 
Signed-off-by: Martin Gaievski <gaievski@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@naveentatikonda naveentatikonda naveentatikonda approved these changes

@Shivamdhar Shivamdhar Shivamdhar approved these changes

Assignees

@martin-gaievski martin-gaievski

Labels
v1.3.0 Issues and PRs related to version 1.3.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.