Support IRSA features for EKS

[aackerman]

This AWS blog post outlines the IAM Roles for Service Accounts feature that is new for EKS.

It appears that the aws-sdk v2 is unlikely to be updated with this functionality and the plugin currently only supports this version. What would be necessary to upgrade to aws-sdk v3?

[AustinTag]

We have an open PR to Logstash for them to use aws-sdk v3 in their plugins (with a version bump). Otherwise, there will be dependency conflicts with logstash plugins if we upgrade to v3.

[JPLachance]

Hello!

A colleague made the required change in a fork of the logstash-mixin-aws plugin.

I used his fork by modifying the Logstash container image:

FROM docker.elastic.co/logstash/logstash-oss:6.8.6

USER root

RUN yum -y install git

USER logstash

RUN echo 'gem "logstash-mixin-aws", :git=>"https://github.com/coveooss/logstash-mixin-aws.git", :branch =>"master"' >> /usr/share/logstash/Gemfile
RUN /usr/share/logstash/bin/logstash-plugin install --no-verify

But then, the installation of the logstash-output-amazon_es plugin fails:

bash-4.4# logstash-plugin install --version 6.4.2 logstash-output-amazon_es
Validating logstash-output-amazon_es-6.4.2
Installing logstash-output-amazon_es
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "aws-sdk":
  In snapshot (Gemfile.lock):
    aws-sdk (= 3.0.1)

  In Gemfile:
    logstash-mixin-aws java was resolved to 4.3.0, which depends on
      aws-sdk (~> 3) java

    logstash-output-amazon_es (= 6.4.2) java was resolved to 6.4.2, which depends on
      aws-sdk (~> 2, >= 2.3.22) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

How complicated is it to bump the aws-sdk version and support assuming a role with web identity in the logstash-output-amazon_es plugin?

I would love to run Logstash in EKS 😄

[itssimon]

Looks like IRSA is also supported in v2 of the SDK now. Would just have to bump version to >=2.11.345 according to this page.

[aackerman]

@itssimon supported in a limited capacity in v2, the CredentialProviderChain which is what this plugin uses to determine credentials doesn't work with web identity credentials that are injected into EKS pods. We need this issue to move forward before updating to aws-sdk v3 in this project.

[aackerman]

We're coming up on the 2 year birthday of this issue next month. Still no support for IRSA from logstash :(