#348: Dependencies: Upgrade Elasticsearch to 8.0.0

[alexklibisz]

Related Issue

Link the related issues as a markdown list.

• Resolves Upgrade to Elasticsearch 8.x #348
• Resolves Deprecate index.elastiknn setting #254

Changes

• Upgrade Elasticsearch from 7.17.6 to 8.8.0
• Upgrade Lucene from 8.11.1 to Lucene 9.0.0
• Upgrade Elasticsearch python client version from 7.17.4 to 8.0.0.
• Make the index.elastiknn setting a no-op and stop using a custom codec. As of 9.0.0, Lucene issue 9843 is resolved, which means the default doc values codec is optimized for best speed. This means Elastiknn can stop overriding the codec. We keep the old custom codecs around for backwards-compatibility. This also resolves Deprecate index.elastiknn setting #254.

Testing and Validation

• Standard CI
• Checked test times and compared to 7.x test times. There doesn't seem to be a meaningful difference. For example, identical 9m26s timing in this job for this PR and in this job for the #392 PR

[alexklibisz]

Holding off on this until #360. I'm hoping that will let us maintain 7.x and 8.x concurrently.

[vladd-bit]

Hello, would it be possible to create an official pre-release of this ? Would like to test it on 8.3.2

[alexklibisz]

You could try this:

• Branch off of this branch.
• Tweak the versions as needed. May or may not need to tweak the code depending on how much has changed from 8.0.0 to 8.3.2.
• Run ./gradlew assembly to build a zip file. Run find elastiknn-plugin -name '*.zip' to find it.
• Install the zip file to your cluster.

[fabriziofortino]

@alexklibisz I see this issue is on hold while waiting for the sbt migration.
Do you have an ETA in mind for when ES8 support could be officially released? Thanks

[alexklibisz]

@fabriziofortino No official timeline. I'm on vacation til Sept 20, so definitely not before then. Tentative plan is "sometime this fall". I'd happily review a PR if someone else wants to try it! Or, if you really want to get on v8, you can build off of this branch as described above.

[alexklibisz]

@fabriziofortino @nfsantos I got the merge conflicts resolved here.

I'm gonna kick off a benchmark job again just to double check there are no big perf regressions. I'm traveling ATM and my macbook air is too slow to run these, so I'm gonna have to wait til I get back home to my desktop.

Would be great if you guys could validate this works for you from the snapshot release. In particular the bit about the index.elastiknn setting.

[nfsantos]

@alexklibisz We have validated the branch with the plugin for 8.4.1, but we cannot validate this one for 8.0. Elasticsearch 8.0 is not working for us because of a problem unrelated to this plugin. So we have to advance to 8.4.1.