Update Elasticsearch client to version 9.x

[l-trotta]

The Elasticsearch java client released a new major version 9.0, introducing a number of breaking changes that will affect this and other spring projects.
I have a draft ready with the minimal changes to make spring-data-elasticsearch work with version 9.0.1 of the client; I'd like to open a PR where we could discuss those and other possible changes, which could be part of the next major release of spring-data-elasticsearch.

Disclaimer: I am one of the maintainer of https://github.com/elastic/elasticsearch-java

[sothawo]

Hi Laura, thank you very much, I really appreciate your support. As we had version 5.5 released yesterday, we now can go forward to integrate Elasticsearch 9 for the next release in autumn.

I hope I'll find the time this weekend to have a look at your changes, and see how this fits in - I'm thinking about backwards compatibility, how to provide the new client, what to do if users were currently providing their own RestClient bean and things like that.

As this is not my daytime job and I got some other hobbies as well, it might take some days...

[l-trotta]

Hello, thanks for the quick reply! And no worries there's no rush :) also there's no plan on doing any other major changes for a while, so this is just early planning.
For now what I did is just the bare minimum to make it work, but there are still some open questions:

1. In version 9 we introduced a new implementation of the RestClient (transport). The legacy one still works, but we'll deprecate it after a while (~version 9.5) since it's based on an already deprecated library (apache http client 4.x). We were thinking that Spring could follow the same approach the client does, so offer both options for users to choose; what I'm not sure about is which one should be the default option. I can update my draft with an initial implementation, but if I understood correctly this particular change would have to be done in the spring-boot repo right?
2. One of the major breaking changes in version 9 involves the Script class: the elasticsearch server accepts not only strings as the Script source, but also SearchTemplates (docs), which in java terms means it can accept either a String or the whole SearchRequest body. Do you think we could add this option in Spring Data too? I can provide an implementation for this, but it will probably introduce breaking changes in the builders.

I will do more testing to see if there are more issues, but all other breaking changes I already took care of in the draft. Let me know if there's more info I can provide!

[sothawo]

So I finally found time. This looks real good for me, please create a PR. You will have to rebase on main and then should notice a failing test, because Spring 7 and Spring Data from the next release on - which will be in November - move to jspecify for nullability annotations. Adaptions there should be easy, the project now is migrated.

I like that this change is basically the minimal step to use ES client 9, there are breaking changes, but that's ok. The documentation needs to be adapted, but with this one, we could already switch to ES 9 on the main branch.

As to your questions:

1. RestClient: We should switch to use the new one as default, but keep the possibility to use the old one (eventually marked as deprecated) making the old one an optional dependency. I do not know how many users of Spring Data Elasticsearch create their own RestClient and pass that into Spring Data Elasticsearch but we need to keep that for at least the next two versions 6.0 and 6.1.
2. The change in the script is something to have a look at. Basically I'd say this must not necessarily be done straight in the beginning, but if we cause breaking changes, then we better do it already when going from 5.5 to 6.0 than when going from 6.0 to 6.1. But there's still time to think about that.

Another thing I came across yesterday (read #3113 for more info): Do you plan to upgrade the client to use jackson 3?

[l-trotta]

Here's the PR! I'll open another one for the new RestClient once I figure out the necessary bom/boot updates ^^"
As for Jackson 3, we haven't tested anything yet, but it seems like it won't be compatible with 2.x, so we'll keep 2.x as default and give users the possibility to switch to 3.x (by providing a dedicated interface). Jackson 3.x as default will probably have to wait until the following major version of the client.

[sothawo]

@l-trotta fyi: I have done some work on the client change to the Rest5Client on a branch in my fork, I'm pretty far, there's still some stuff to do there, but next week I want find more time, and then I'm on vacation for two weeks. Just to let you know, no need to do anything there.

[l-trotta]

@sothawo Thank you! Just to clarify, is there anything else I should do in the spring-boot repo to consolidate the version change, or is it part of your changes?

[sothawo]

No, there is currently nothing for you to do thanks for offering your support; I did some more work yesterday evening and am getting to a version where the default will be Rest5Client and users are still able to use the RestClient