Adding assume_role_with_web_identity

[cdenneen]

Leaving aws-sdk v2 but adding the change by @coveooss

Hoping for faster merge.

Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/

[cdenneen]

@jsvd @andsel would you mind reviewing?

[jsvd]

hi @cdenneen thanks for your effort in this, and apologies for the long long time it took to act on these plugins.

I spent some time last week working to facilitate the transition to aws-sdk and help avoid having to import the top level ask-sdk v3 gem that brings >300 dependencies through the aws-sdk-resources gem. For this I joined all aws plugins in an integration plugin (maintaining history and contributor attribution) and then created a pr [1] that moves all the plugins to the v3 sdk using the service-specific leaf gems + aws-sdk-core.

This code is passing all tests we had before, and I've confirmed manually the s3, sqs and sns plugins work.

The next step is to make sure the docs won't break our documentation due to bad links, do a final review and publish the aws integration plugin.
Once this is done we can much more easily add a feature like this one as it's done in a single repository.

I do apologise again as you've gone through the work of doing this change in individual repos but hopefully the above shows that we're moving the integration forward and soon we'll be able to add more features on top of it.

[0] https://github.com/logstash-plugins/logstash-integration-aws/tree/bootstrap_work
[1] logstash-plugins/logstash-integration-aws#6

[cdenneen]

@jsvd thanks. When can we expect the updates? Are core and core plug-in api still extremely outdated? I did get the changes working and have validated in my life be setup that sqs is properly working.
I do have to ask that there needs to be better communication, response and feedback from the @elastic team. There are issues, prs and comments for extremely long periods of time.
There should be public forum for discussion around these things and sadly I would say elastics “discuss” is never responded to and threads are closed.

Maybe community slack?

Between the 7 projects here this is the first response I’ve gotten but there are others waiting years!

Thanks again

[christianherweg0807]

@jsvd
Just a short question: What is the problem with the 300+ dependency´s? In the past aws-sdk ruby was a huge mono gem or missed i something?
I think it is a great plan to move all those aws things into one plugin, but is it possible to allow updating of aws-sdk by
using
s.add_runtime_dependency 'aws-sdk', '>= 2' instead of s.add_runtime_dependency 'aws-sdk', '>~2'

This would make it possible for all other AWS Logstash plugins to upgrade (if they need).
In my special case it is not possible to run logstash an AWS k8s, because it´s not supported by sdk-v2.

regards
Christian

[jsvd]

Just a short question: What is the problem with the 300+ dependency´s? In the past aws-sdk ruby was a huge mono gem or missed i something?

@christianherweg0807 It's a fair question, it's mostly about keeping track and vetting a lot of new individual dependencies and code (v2 was big but only received minor fixes lately). The aws-sdk v2 is about 27MB in size, while aws-sdk v3 is 169MB. Also, it causes Bundler to take longer to resolve the lock file, do any plugin manager tasks, impacting restart and startup times.

Are core and core plug-in api still extremely outdated?

@cdenneen we stopped producing these gems for a while now as they didn't really reflect the boundary they were supposed to, and were an artifact of a time where release cadence for Logstash wasn't known, so we needed to decouple releases from API. Logstash now has minor releases about every 3 months and we're able to better express compatibility through mixin adapters when necessary.

I do have to ask that there needs to be better communication, response and feedback from the https://github.com/elastic team. There are issues, prs and comments for extremely long periods of time.

Agreed, there are many many parallel threads of issues and contributions that deserve an answer for a long time. Believe it or not, since about a year ago we started doing more tracking of a lot of these and we're making progress.

As for the aws-sdkv3 in particular I'm driving it to completion as fast as possible, report back here when I have an update.

[cherweg]

@jsvd : Could you please make an update possible, while we are waiting for the new plugin by simply using:

s.add_runtime_dependency 'aws-sdk', '>= 2'

in the dependency’s?

thank you
Christian