Fargate Log Driver Support v2

[abby-fuller]

Fargate Log Driver Support v2 (fluentd, gelf, syslog). We are also building a log router that can 1) Forward logs to AWS services such as Cloudwatch, Amazon Elasticsearch, S3, Amazon Managed Streaming for Kafka and Kinesis Analytics 2) Extensible to partner destinations through Fluentd or Fluent Bit output plugins 3) Filter on patterns in the logs stream, eg send http 200 to S3 vs http 400,500 to Elasticsearch

[abby-fuller]

Referencing an existing issue from ecs-agent: aws/amazon-ecs-agent/issues/734

[agurha]

whats the status on this?

[m-barthelemy]

Hi,

Really interested in knowing the status of this too.
Only offering awslogs as a log driver is... unbelievable

[masterxavierfox]

Hello Any progress on this?

[arvinep]

Is this feature somehow scheduled for any release? In which release we can expect it?

[debu99]

any update

[jamiegs]

Will this include sumologic log driver support?

[idohalevi]

can this include Logz.io as well?

[inductor]

Datadog support please :(

[chungath]

Any update on the FluentD log driver?

[btotharye]

bump for fluentd support, kinda amazing you only really support awslogs and splunk on this, seems to be a showstopper for a lot using ECS Fargate that don't want to pay for cloudwatch.

[mhausenblas]

Update: getting closer. Check out the launch of the AWS Fluent Bit plugins and what you can do with it.

[jritsema]

@mhausenblas Thank you for the docs!

Looking ahead, we are also working on a feature to further simplify installing and configuring fluent bit plugins on AWS Fargate, Amazon ECS, and Amazon EKS. You can follow this feature via the Issue 10 of our AWS container roadmap.

Does this mean that the ECS setup in the doc (fluentbit as a sidecar and the fluentd log driver config) won't work for fargate yet until this issue is resolved? Thanks.

[akshayram-wolverine]

hey @jritsema we are shipping the full support for Fluent Bit support on Fargate and working with partners to make sure the interface is extensible. We will update you when we have a preview in place to try out Fluent bit on Fargate which should be extensible to multiple AWS and partner destinations. Meanwhile, the AWS for Fluent Bit image can be used right away as a daemon set or a daemon service on ECS/EC2 and EKS/EC2 clusters

[btotharye]

@akshayram-wolverine can you let me know when fargate is ready here at Pearson we really need this

[idohalevi]

Why Fluentbit and not FluentD?

[chrisandrewcl]

Really looking forward to this. Is there any news?

[akshayram-wolverine]

@inductor datadog support through here: fluent/fluent-bit#1502. @idohalevi our tests show that fluentbit is lighter in resource consumption. We will also support Fluentd as we recognize there is already a large community of fluentd plugins. @chrisandrewcl we will update this thread as soon as we have a preview in place.

[akshayram-wolverine]

We heard for customers asking for extensibility for log destinations on Fargate. We now have a developer preview that allows customers to use a log router called Firelens that works with Fluentd and Fluent Bit to route logs to AWS services such as Cloudwatch, Amazon Elasticsearch, S3 and partner tools you can use through the output plugins on Fluent Bit or Fluentd.

See here: https://github.com/aws/containers-roadmap/tree/master/preview-programs/firelens

[raskad]

@akshayram-wolverine
I tried this new feature today. First of all thanks for the great work!
My first two experiments where with Fluent Bit, routing to Kinesis Firehouse and AWS Elasticsearch. Both worked without much trouble. I really like, that I can configure everything in the taskdefinition and dont have to build a custom docker image for logging.

Possible enhancements:
A great enhancement would be settings to enrich the ECS Metadata. It would be really nice to add custom data fields. You could apply these similar to the "exclude-pattern" and "include-pattern" fields in the firelens options, or alternativeley in the "firelensConfiguration".
My usecase is that our taskdefinition revisions do not always correspond with the application version numbers. Adding a static field e.g. "app-version": "v1.1.14" or a field based on an environment variable e.g. "app-version": "$VERSION" woud make easier analysis possible. But I am sure others have different usecases for this aswell.

[binarylogic]

@akshayram-wolverine thanks for the updates! A humble request: would it be possible to also support Vector? We built Vector specifically for this purpose. We use Fargate internally and our use case requires Vector for the following reasons:

1. Performance and correctness.
2. The Vector aws_cloudwatch_logs sink supports dynamic partitioning. It appears that the Fluentbit Cloudwatch Logs plugin lacks support for this.
3. The Vector aws_s3 sink. Also, supports dynamic partitioning.
4. Finally, metrics. Specifically the Vector log_to_metric transform. Our apps emit an event stream and we use Vector to reduce events into metrics and forward them via the prometheus sink.

There are more reasons, but in general, it'd be great to make this feature open so users could choose the best solution for their use case.

Thanks again for your work on this feature 😄 . It's exciting to see progress.

[annuh]

Thanks for this new feature, it works great without much configuration!

I would like to forward our logs to multiple destinations, like AWS Cloudwatch and Datadog. Is this possible with Firelens?

[nbrandaleone]

The local ECR repos are not working for the Firelens container. Only the Github repo is accessible. I suspect it to be a cross-account IAM permissioning issue.

$ ecs-cli pull --region us-east-1 --registry-id 906394416424 aws-for-fluent-bit:latest
FATA[0000] Error executing 'pull': failed to serialize authorization token: ecr: Failed to get authorization token: UnrecognizedClientException: The security token included in the request is invalid.
status code: 400, request id: c5aa805d-f457-4103-94e6-e24ce368a269

[PettitWesley]

@annuh Fluent Bit has the ability to route logs from one source to multiple destinations; see here: https://fluentbit.io/documentation/0.12/getting_started/output.html

During the preview, FireLens only supports a single destination. Fluent Bit & Fluentd can do a lot of things which FireLens does not currently support. Without spoiling too much, after the preview FireLens will allow you to do a lot more. Including route to multiple destinations. Once that's announced we look forward to your feedback on the experience :)

[PettitWesley]

@nbrandaleone The ECR repositories are accessible; I've verified this myself. Unfortunately, the URIs in our documentation were wrong. Apologies for this.

#477

UnrecognizedClientException: The security token included in the request is invalid.

For the ECS CLI error that you got, this is caused when your credentials are expired.

[akshayram-wolverine]

@raskad thanks for the feedback! We do have some of the capabilities that you asked for as we come out of preview.

@binarylogic very interesting thanks for sharing! We see Fluent Bit and Fluentd as communities that have extensive plugin support and developers who understand fluent.conf. If we hear from customers that Vector (events ->metrics is cool!) is an alternative we are open to adding it as a router type. My email is akshram@amazon.com, I would love to learn more about Vector and how it simplifies logging for customers.

[tedder]

I'm especially interested in log drivers that don't involve a sidecar. That's one advantage of cloudwatch logs- it requires very little configuration. Of course it has downsides, including adding a lot of cost if you aren't simply leaving it in cwlogs. It appears only Splunk and cwlogs are available without sidecars and there's no plan for otherwise.

[akshayram-wolverine]

@tedder thanks for the feedback. We did think hard about continuing with in tree integrations where drivers are built in to the container runtime. What we observed was while there are upsides such that you don't see a sidecar, there are cons such as correlation of failure when there is a lot of backpressure and coupling update cycles of partners and underlying compute. That's also why we went with Fluent Bit as a recommended approach as its light, and independently observable. We will sometime over the next year think through a concept of hidecars where we hide the aws for fluent bit image from customers to make it even simpler. We have the under the hood and resource footprint here: https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/ and jeff bar here: https://aws.amazon.com/blogs/aws/announcing-firelens-a-new-way-to-manage-container-logs/. supported partners: https://aws.amazon.com/fargate/partners/?nc=sn&loc=5

[lkdmid]

So, still no GELF?

[arliber]

Update: I solved this by going another direction - sending my logs using AWS FireLens.
I'm using Coralogix for logs so they have nicely documented this solution:
https://coralogix.com/integrations/aws-ecs-ec2-fargate-logs/