TCP Client to ES Deprecated - Switch to HTTP

[bacoboy]

The elastic4s TCPClient you are using to communicate with ElasticSearch has deprecated the TCPClient in favor of the HTTPClient. It will be removed in the next major version of elastic4s which already has a first release candidate.

According to the README they say you can just swap out the import paths from elastic-tcp to elastic-http and change code uses of TCPClient to HTTPClient. That said, there is a warning about the responses being different from client.execute() so the change is a little more involved:

Requests are the same for either client, but response classes may vary slightly as the HTTP response classes model the returned JSON whereas the TCP response classes wrap the Java client classes

Once complete and added to TheHive, people can insert an aws authenticating proxy to use AWS Hosted ElasticSearch ending a standoff between TheHive developers and AWS dogma. Apparently support for a signed http transport in elastic4s is in the works.

Ideally a pluggable transport would allow people to pick and choose (tcp, http, http-aws) rather than be dictated to them.

Anyway this isn't about AWS ES but the TCP transport going away in the library you are using. Better to start a discussion about what to do and when than wait until the code atrophies.

[To-om]

Hi @bacoboy,

Several week ago, I've tried to add support of elastic4s HTTP client to Elastic4play but some operations were not supported. I've updated elastic4s version and it works better but there are many problems to support both TCP and HTTP.
The main problem is that the result of an operation (a search for example) doesn't return the class. TCP client returns a com.sksamuel.elastic4s.searches.RichSearchResponse whereas HTTP client returns a com.sksamuel.elastic4s.http.search.SearchResponse and conversion between the two types is not possible (they doesn't contain the same data).
In order to support HTTP and TCP, I must add an abstraction layer on operation results.

For you information, we have many problems with ElasticSearch and we seriously think about migration to another database technology. But migration to a new database is not an easy task and will take time.

[bacoboy]

Yes, because you are using the the result classes directly you would have to create an abstraction layer anyway -- in which case you could support a plugable persistence layer -- including a standard database.

[dcode]

@To-om does 5e2e12b fully implement the HTTP transport, or is there more to do. I'm ultimately keen to update TheHive to use an ES 6.x backend, but it seems the TCP transport and Netty dependencies are the stopper for that. Maybe there's other issues, but I thought I'd ask.

I'm happy to test and provide useful feedback. I use Elastic extensively with the @rocknsm project and I'm looking to update my entire cluster to 6.x.

Thanks!

[slatemine]

+1
@To-om does 5e2e12b fully implement the HTTP transport
This issue, we're in the process of spinning up a production cluster and were going with AWS Elasticsearch just to provide a quick low maintenance ES cluster ( daily back up etc ). Anyhow AWS does not support TCP clients.

I see the branch on which this work was done stopped, was it successful or a dead end.

[To-om]

From version 1.11, HTTP communication is used. TCPClient is not used any longer.