elastic version/license check in elasticsearch client library

[rursprung]

it seems that elastic has started adding license checks to their client libraries, e.g. elastic/elasticsearch-py#1623 or elastic/elasticsearch-net#5728.

i haven't seen the same yet for the java client, but i also didn't look for it yet (because the java client is a bit special as it's part of the elasticsearch repo, making it hard to track what's going on there).

can you please ensure that you're not upgrading to a library version from elastic which has this check at least until the refactorings from #1853 are done and a first OpenSearch integration has been done as well (#1770 has been closed and there's currently no open ticket to track this - maybe it would make sense to re-open it?)?
if i understood correctly these new license checks might also affect some people using Elasticsearch, depending on which version and which license they're using.

[sothawo]

Looking at that code, it is not a license check, but a check that the Elasticsearch client library that is used is working against an Elasticsearch cluster and that the client version and the cluster version match (however this matching is defined), and this is changed in the Java code as well as far as I could see. It's not our task to judge the decision to use such a check.

But for Spring Data Elasticsearch it has always been a potential error source if the client library that is used at runtime is different than the one that was used to build the Spring Data Elasticsearch library and different from the version that the cluster is running. To be able to find errors due to some version mismatch, Spring Data Elasticsearch is logging all three versions when first request is done - this is completely independent from what Elasticsearch does with these code changes, this looks like this in the current main branch:

2021-07-23 18:29:08.949  INFO o.s.d.elasticsearch.support.VersionInfo  : Version Spring Data Elasticsearch: 4.3.0-SNAPSHOT
2021-07-23 18:29:08.950  INFO o.s.d.elasticsearch.support.VersionInfo  : Version Elasticsearch Client in build: 7.13.3
2021-07-23 18:29:08.950  INFO o.s.d.elasticsearch.support.VersionInfo  : Version Elasticsearch Client used: 7.13.3
2021-07-23 18:29:08.950  INFO o.s.d.elasticsearch.support.VersionInfo  : Version Elasticsearch cluster: 7.13.3

What does these checks by Elasticsearch mean for Spring Data Elasticsearch? Currently there is no problem with that because Spring Data Elasticsearch is targeted, built and tested against a defined version of Elasticsearch, these are documented, the current development is using 7.13.1.

The license for the RestHighLevelClient form Elasticsearch is changed as well, but it can be used until Elasticsearch releases the new Java Client they are working on (https://github.com/elastic/elasticsearch-java).

We have started to refactor the architecture of how Spring Data Elasticsearch accesses the cluster and we will change the code to support multiple Connectors (that's how we probably will call them). On of theses connectors will then be using the new Elasticsearch client. Another one could be based on the 7.10.2 RestHighLevelClient or on some client library provided by Opensearch.

What about using Spring Data Elasticsearch with Opensearch? The first test I did failed on the very first call I tried to make: getting the cluster info (call to the default endpoint of GET /). This did not work with an Elasticsearch client in version 7.10.2, because Opensearch had removed the tagline from the cluster response, which had always been a mandatory field in the answer. After I reported that they seem to have put it in again, I have not tested that yet.

We currently don't know - and have no time to deep dive into this - how compatible an Opensearch cluster is with Spring Data Elasticsearch. As said, we work on changing the architecture to support both cluster providers. But as long as Opensearch does not provide their own version of the org.opensearch.client.RestHighLevelClient as an artifact on maven or some other public repository, I cannot use both versions (7.13.x and 7.10.2) in the same project.

The forked codebase does contain the sourcecode for the RestHighLevelClient- https://opensearch.org/docs/clients/java-rest-high-level/, but I am not aware of a released version of the client library. The Opensearch documentation explicitly talks about using the Elasticsearch library (https://opensearch.org/docs/clients/java-rest-high-level/).

So the roadmap for us is something like:

• refactor the access to a cluster by adding connectors for different implementations
• implement a connector using the current RestHighLevelClient to verify the connector interface
• when Elasticsearch releases the new client, provide a connector that uses that and remove the existing non-connector based code that accesses the cluster
• change the connector that uses the RestHighLevelClient to use ES 7.10.2, so it might be compatible with Opensearch
• if Opensearch provides a client implementation of their own, integrate that instead of using the old ES dependency.

So we have an eye on that, we are waiting for Elasticsearch to release their new client and looking if Opensearch releases a client of their own but until that we will keep on to develop againts the current Elasticsearch version.

[wu-sheng]

FYI, we(from Apache) have submitted a request to the OpenSearch community to continue their own Java high-level client. https://discuss.opendistrocommunity.dev/t/elasic-java-clients-are-no-longer-open-source/6470/9

[sdavids13]

FYI AWS blog post saying OpenSearch will be publishing their own client: https://aws.amazon.com/blogs/opensource/keeping-clients-of-opensearch-and-elasticsearch-compatible-with-open-source/

[sothawo]

What's missing on this list is a client for the current versions. The mentioned repository for a Java client is the new client from Elasticsearch which has not been yet released.

[sothawo]

As outlined in #1918 we will add the possibility to include an OpenSearch client library when the new Elasticsearch client is available and Spring Data Elasticsearch is internally refactored. In the meantime we upgrade the used Elasticsearch client libraries to 7.14.
This upgrade is done #1933, so I'm closing this issue