Support for Secured AWS ElasticSearch

[lostinplace]

Is there a way (or a plan) to support IAM secured clusters on AWS ElastiscSearch?

[sksamuel]

What does IAM require?

…

On 14 Feb 2017 18:44, "Chris Wheeler" ***@***.***> wrote: Is there a way (or a plan) to support IAM secured clusters on AWS ElastiscSearch? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#749>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAtZGpYfyCrFdTAvSrnwMaOfXRbd3iOQks5rcfYAgaJpZM4MA1_2> .

[lostinplace]

You have to sign the requests with your AWS credentials

see https://aws.amazon.com/blogs/security/how-to-control-access-to-your-amazon-elasticsearch-service-domain/

[sksamuel]

If its an Amazon specific thing then I don't think it should be part of a general e4s project.

…

On 16 February 2017 at 20:32, Chris Wheeler ***@***.***> wrote: You have to sign the requests with your AWS credentials see https://aws.amazon.com/blogs/security/how-to-control- access-to-your-amazon-elasticsearch-service-domain/ — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#749 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAtZGhuyz_zDRdgt2pIgaoPPoRZnIGanks5rdLJ7gaJpZM4MA1_2> .

[lostinplace]

does that mean that if I were to submit a pull request that did this, it would be rejected?

[sksamuel]

It really depends on what the support is for. If its something unique to elasticsearch then it would be fine to add to elastic4s. If it's a generic authentication used by all of Amazon's services then its better in some amazon library no?

[sksamuel]

Do you think this would work, if I add a hook into the HTTP client to use it easily?
https://github.com/inreachventures/aws-signing-request-interceptor

[lostinplace]

yup, that's exactly what's needed

[mdarcemont]

I'm very interested in a hook into the HTTP client too.

[alex1712]

I hope you already got it to work, but I use the same principle as the apache interceptor described previously and it works for me.

The only difference is that I had to modify the Host header to remove the port.

Here is the code -> https://gist.github.com/alex1712/f41091ea316b6cdb0429dda71151a6d9

Maybe can help someone.

[lustefaniak]

Would be great if something like that was provided in elastic4s. Probably as additional module.

[dmitry-avershin]

hi @sksamuel, are you planning to add a hook to the http client that will ease the integration with AWS or any other service that requires signing the request in future versions of elastic4s?

[sksamuel]

@dmitry-avershin I'll add something into v6.

[dmitry-avershin]

@sksamuel nice, I can support here if needed

[sksamuel]

@dmitry-avershin feel free to create a PR. I would be looking at doing something along the lines of the link I posted in this thread, or open to other ideas.

[nemo83]

Hey all, I'm interested in this feature too. Is the hook implemented on any branch? Or shall I temporarily implement the code as per @alex1712? Please feel free to ping me if you need anyone to test it on AWS !

Great work everyone!

[sksamuel]

Yep please have a go.

[nemo83]

Ok, I did a test with @alex1712 snippet and elastic4s version "5.4.3" and it works like a charm.

I can work on a PR if no else is working on it....

[sksamuel]

I'm not working on it personally, there's been no movement from others so I think it's safe to pick up.

[saraiva132]

Would also love to have this. @nemo83, are you working on a PR?

[nemo83]

unfortunately not :-( @saraiva132

[saraiva132]

@alex1712 gist seems to contain everything I need which is cool. However, the dependency to aws-request-signer worries me. I will look into it as well and If I figure a decent enough solution I might look if it is enough for a PR.

[alex1712]

Hi, thanks for taking care.

This depency is actually this project here :
https://github.com/ticofab/aws-request-signer

Is only one class if you want a check the code.

[saraiva132]

@alex1712 Yeah, I have read the code and am also reading the aws documentation for signing (version 4). I am working on a request-signer and an object to provide a client with the interceptor and afterwards I will try and make a PR to see if this can be added to elastic4s 6 as a module. So basically the same as your gist and that project.

[sksamuel]

What I think we need is an elastic4s-aws module which contains an apache http interceptor. This interceptor would use the aws signer to sign the requests. Then the user can add the interceptor to the HttpClient, and we can provide an AwsHttpClient object method to do that conveniently.

[saraiva132]

@sksamuel Yep. Will work on just that.

[saraiva132]

@sksamuel I just finished my first working draft (took me longer than expected. Going to make the PR with my proposal soon, any advice from you? (i.e. expected tests, code style)

[sksamuel]

I'm not fussy - as long as there's tests for it, and it works, I'm happy :)

[sksamuel]

Merged into 6.0

[uccmen]

How do I use this? I see no documentation on README or wherever.

[hekaldama]

@uccmen did you get an answer figure it out?

[uccmen]

@hekaldama - I got it to work with the below conf.

val elasticEndpoint = s"elasticsearch://$esHost:$esPort?ssl=true"
val esConfig = Aws4ElasticConfig(elasticEndpoint, awsAccessKey, awsSecretKey, awsRegion)
val esClient = Aws4ElasticClient(esConfig)

and sorry for the late response!