AWS OpenSearch / ElasticSearch compatibility

[jvmlet]

Hello
Given the recent development with ElasticSearch fork, what would be spring-data-elasticsearch roadmap/preferences ?
The day that rest client API become incompatible with OpenSearch is not that far.
Are you going to support both servers (ES and OS) / only one of them (if yes - which one)?

[mp911de]

The recent development put the entire community in a difficult situation. It's common practice for data stores to implement an additional communication protocol besides the native one (Apache Geode providing Redis as protocol, AWS DocumentDB with MongoDB protocol support, and many more) which enables the use of existing client-side libraries. The flip side however is that certain assumptions of those connectors are rendered invalid because the remote side behaves differently than expected.

Spring Data Elasticsearch heavily relies on Elasticsearch client libraries. If OpenSearch decides to change the Client API that isn't compatible with Elasticsearch then Spring Data Elasticsearch could not integrate with that particular feature or break entirely.

Since future development isn't clear at this point and this project is driven by the community, it's always good to get involved and help the project to drive its evolution.

[jvmlet]

The compatibility might also be broken at the point ES decides to expose additional methods over rest in future versions (and DATAES integrates them ). Invoking these endpoints will definitely won't work if executed against OS server.

[mp911de]

Indeed, that could break OS users. OpenSearch was never a goal of this project, frankly speaking. In an extreme case, one would have to either fork this project and adapt it to OpenSearch or contributors would help to rewrite the codebase so that the underlying transport/API/… would be agnostic.

In any case, we will have to see what the future holds.

[sothawo]

For the time being we should watch how OpenSearch evolves. As long as the REST APIs are compatible, there is no problem for Spring Data Elasticsearch sticking to ES client libs. Problems will come when there is a functional or API drift between the two. We then must decide to either support only one, or add an abstraction layer above both - which would mean that we could only support the common feature set. This could get ugly.

But for the time being, we stick to ES and have an eye on what is going on with OS. First thing I think would make sense is wait for a docker image to be available for OpenSearch and use that to run the integration tests against.

[sothawo]

I think we can close this issue. Currently we cannot do more than monitor how Opensearch evolves.