Bump AWS SDK version to version "~>3".

[AustinTag]

We have a plugin:

https://github.com/awslabs/logstash-output-amazon_es

which we want to bump the aws-sdk version to "~>3". To avoid any dependency conflicts with other logstash plugins, we would also like to bump the version of aws-sdk used by logstash-mixin-aws. My thinking is that this should be a major version bump, and that each of the logstash plugins that are using "logstash-mixin-aws" as a runtime dependency should upgrade their dependency to "logstash-mixin-aws" with a version bump as well.

I am in the process of developing and testing this change, but wanted to create the issue to track any discussion about this.

[AustinTag]

Actually, after making this change locally and testing all of the logstash plugins that use logstash-mixin-aws as a dependency, it appears that upgrading to aws-sdk version "~> 3" is not a breaking change for any of these plugins (that is, all of their tests passed with the upgraded sdk). Therefore it seems safe to simply do a minor version bump in logstash-mixin-aws. Thoughts?

[colinsurprenant]

As mentioned in #39 bumping this dependency version has the side effect of adding ~200 more dependencies and we will defer using v3 of the aws-sdk to our upcoming aws integration plugin (see https://www.elastic.co/blog/logstash-lines-introduce-integration-plugins) where we will be in a better place to only pick the required dependencies.

[HitkoDev]

@colinsurprenant v3 also brings support for web identity provider, which is a crucial security feature when running on AWS EKS.

[AustinTag]

Is there an ETA on the AWS integrations plugin?

[colinsurprenant]

We are actively moving forward with integration plugins but we don't have an ETA.

[JPLachance]

Hello @colinsurprenant,

A colleague did the aws-sdk upgrade and added the support for AssumeRoleWebIdentityCredentials in a fork: master...coveooss:CLOUDINFRA-2050---logstash-in-EKS

Do you want us to submit a pull request?

Thanks!

[colinsurprenant]

@JPLachance Sure! Go ahead! Thanks.

[itssimon]

It looks like bumping the AWS SDK version to at least 2.11.345 would introduce support for IAM roles for service accounts (IRSA). That should be an even smaller change to make, but add heaps of value for AWS EKS users.

Would you consider this as an interim solution?

[dgonzalezruiz]

This has been opened for over a year now 😅 is there any chance we can get upgraded aws-sdk gem soonish? As @itssimon pointed out, latest v2 should already support temporary creds. issued via AssumeRoleWebIdentityCredentials method, which would unblock using logstash with any AWS related plugin, that wants to benefit from reduced complexity handling security credentials (thanks to IRSA).

Logstash integration plugins seem great, but it's been over a year and there's only three of them available (seem like that the same ones made available in the original announcement). What's the best course of action that one could take here @colinsurprenant ?

[dgarbus]

This has been opened for over a year now 😅 is there any chance we can get upgraded aws-sdk gem soonish? As @itssimon pointed out, latest v2 should already support temporary creds. issued via AssumeRoleWebIdentityCredentials method, which would unblock using logstash with any AWS related plugin, that wants to benefit from reduced complexity handling security credentials (thanks to IRSA).

Logstash integration plugins seem great, but it's been over a year and there's only three of them available (seem like that the same ones made available in the original announcement). What's the best course of action that one could take here @colinsurprenant ?

I tried to get this working with v2 of the ruby SDK and had no success. I don't even see a reference to AssumeRoleWithWebIdentity in the v2 source code.

[overskylab]

any updates?

[aackerman]

v2 of the SDK doesn't doesn't support web identity credentials through the CredentialProviderChain so v2 isn't really an option since that's what most open source projects use. The 6.8.13 logstash docker image with the logstash-output-amazon_es plugin uses v2.11.540 and doesn't work with IRSA even though aws-sdk-core is above the documented minimum SDK version.

[karthik-devo]

Any updates on this issue to support IRSA?

[christianherweg0807]

I´ve tested with aws sdk v3, by patching mixin locally...
Everything works fine. @colinsurprenant : the wide range of new dependency comes with the "new" modularity of aws-sdk.
There is a versioned gem for ervery service now.

The pessimistic version locking in this plugin block´s ervery plugin developer from using the new SDK. So please update or change the dependency to >= 2.

[christianherweg0807]

We´ve a manually patched version in our test envirnonment. Everything works as expected with aws-sdk 3.

regards
christian

[christianherweg0807]

@roaksoax :
Is it possible to help on releasing this fix?
The SDK 2 is really old and without security patches since 11/2021...please help out!

thank you
Christian

[cherweg]

Any news on this?

[cdenneen]

@andsel @colinsurprenant Should this project be considered abandoned and all other projects should be migrated directly to use the sdk?

[christianherweg0807]

FYI:

declare -a PluginList=("logstash-input-s3-sns-sqs" "logstash-input-s3" "logstash-input-sqs" "logstash-output-s3" "logstash-output-sns" "logstash-output-sqs" "logstash-output-cloudwatch" "logstash-mixin-aws")
LOGSTASH_HOME=${LOGSTASH_HOME:-'.'}
for plugin in ${PluginList[@]}; do
   echo "Try uninstalling plugin ${plugin}"
   ${LOGSTASH_HOME}/bin/logstash-plugin remove $plugin
done

./bin/logstash-plugin install --version 0.1.0.pre logstash-integration-aws 

0     logstash-integration-aws (0.1.0.pre)                                                                                                                                                  
561       aws-sdk-cloudfront                                                                                                                                                                  
562       aws-sdk-cloudwatch                                                                                                                                                                  
563       aws-sdk-core (~> 3)                                                                                                                                                                 
564       aws-sdk-resourcegroups                                                                                                                                                              
565       aws-sdk-s3                                                                                                                                                                          
566       aws-sdk-sns                                                                                                                                                                         
567       aws-sdk-sqs                                                                                                                                                                         
568       concurrent-ruby                                                                                                                                                                     
569       logstash-codec-json                                                                                                                                                                 
570       logstash-codec-plain                                                                                                                                                                
571       logstash-core-plugin-api (>= 2.1.12, <= 2.99)                                                                                                                                       
572       rufus-scheduler (>= 3.0.9)                                                                                                                                                          
573       stud (~> 0.0.22) 

[cdenneen]

you can't remove logstash-mixin-aws this way:

Step 9/12 : RUN /usr/share/logstash/bin/logstash-plugin remove logstash-mixin-aws
 ---> Running in 261b383fb44c
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
ERROR: Operation aborted, cannot remove plugin, message: This plugin has not been previously installed
The command '/bin/sh -c /usr/share/logstash/bin/logstash-plugin remove logstash-mixin-aws' returned a non-zero code: 1

Here is currently the Dockerfile:

ARG logstash_version
FROM docker.elastic.co/logstash/logstash-oss:${logstash_version}

RUN /usr/share/logstash/bin/logstash-plugin remove logstash-input-s3
RUN /usr/share/logstash/bin/logstash-plugin remove logstash-input-sqs
RUN /usr/share/logstash/bin/logstash-plugin remove logstash-output-s3
RUN /usr/share/logstash/bin/logstash-plugin remove logstash-output-sns
RUN /usr/share/logstash/bin/logstash-plugin remove logstash-output-sqs
RUN /usr/share/logstash/bin/logstash-plugin remove logstash-output-cloudwatch
RUN /usr/share/logstash/bin/logstash-plugin remove logstash-mixin-aws


RUN /usr/share/logstash/bin/logstash-plugin install --version 0.1.0.pre logstash-integration-aws
RUN /usr/share/logstash/bin/logstash-plugin install logstash-input-kinesis
RUN /usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch

[cdenneen]

Seems like if you don't remove logstash-mixin-aws it will still install the new preview integration-aws:

Step 9/11 : RUN /usr/share/logstash/bin/logstash-plugin install --version 0.1.0.pre logstash-integration-aws
 ---> Running in f07aa3d08bdc
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-integration-aws-0.1.0.pre
Resolving mixin dependencies
Installing logstash-integration-aws
Installation successful

[christianherweg0807]

Please do a

grep aws Gemfile.lock 

to see the installed sdk version.

[andsel]

Hey @cdenneen and @christianherweg0807 thanks for your interest in this upgrade, please note that's some effort in upgrading to AWS v3 is also happening at https://github.com/logstash-plugins/logstash-integration-aws/pulls

[christianherweg0807]

sure for this it is a prerelease.
But better a prerelease, than unable to run on k8s...

This is a real big production issue for us.

regards
Christian

[jsvd]

@christianherweg0807 we're getting close to getting the aws integration out, so it'd be nice to push a 0.1.0 release to rubygems. Can you hand over the ownership in rubygems.org to rubygems@elastic.co ?
We can also use a different name if you prefer to keep "logstash-integration-aws" (first come first served :) )

[cherweg]

Sure. I’m today afk. But tomorrow, I’ll handover. Regards Christian Von meinem iPhone gesendet

…

Am 06.07.2022 um 11:24 schrieb João Duarte ***@***.***>:  @christianherweg0807 we're getting close to getting the aws integration out, so it'd be nice to push a 0.1.0 release to rubygems. Can you hand over the ownership in rubygems.org to ***@***.*** ? We can also use a different name if you prefer to keep "logstash-integration-aws" (first come first served :) ) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[christianherweg0807]

@jsvd :
elastic was added as an unconfirmed owner. Ownership access will be enabled after the user clicks on the confirmation mail sent to their email.

Plz confirm the access, so I could remove myself

regards
Christian

[jsvd]

Confirmed, many thanks!

[jsvd]

With the logstash-integration-aws released which includes aws-sdk-v3 (logstash-plugins/logstash-integration-aws#9), we can close this issue. Logstash 8.4.0 will be released soon, an include the integration plugin by default.