out_cloudwatch_logs: add support for record accessor

[dahu33]

Is your feature request related to a problem? Please describe.

I would like to dynamically add the Kubernetes namespace in the Cloudwatch Logs group name. This is useful to:

• restrict user log access to certain namespaces.
• log ingestion monitoring (Cloudwatch Logs metrics are per log group name).

Describe the solution you'd like
I believe the solution would be to implement record accessor in the out_cloudwatch_logs plugin. That way the user could freely name its log groups and log streams based on record information.

Example:

[OUTPUT]
    Name cloudwatch_logs
    Match   *
    region us-east-1
    log_group_name /aws/containerinsights/${CLUSTER_NAME}/namespace/$kubernetes['namespace_name']

Describe alternatives you've considered
Is there any?

[dahu33]

@PettitWesley @edsiper would you mind having look at this?

[PettitWesley]

I believe the solution would be to implement record accessor in the out_cloudwatch_logs plugin. That way the user could freely name its log groups and log streams based on record information.

@dahu33 I believe you are correct

We built a templating feature in the cloudwatch go plugin: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit#templating-log-group-and-stream-names

It would be cool to implement it in the cloudwatch_logs plugin too, but I am not prioritizing it right now since we already have the go plugin templating implementation.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[gabegorelick]

It would be cool to implement it in the cloudwatch_logs plugin too

Is the eventual goal to implement record accessor in cloudwatch_logs, or to port the templating feature of cloudwatch into cloudwatch_logs?

aws/amazon-cloudwatch-logs-for-fluent-bit#131 (comment), mentions that cloudwatch didn't use record accessor because it's not available to Go plugins, but that's not an issue for cloudwatch_logs.

[PettitWesley]

Is the eventual goal to implement record accessor in cloudwatch_logs

Yes. Record accessor is the Fluent Bit core way of doing templating. So we would use that method in cloudwatch_logs. The templating feature in the cloudwatch plugin works the way it does because it uses Go's templating library which unfortunately doesn't easily support the same syntax as record accessor.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[geckofu]

FWIW and to keep the issue open, aws-for-fluent-bit has both out_cloudwatch and out_cloudwatch_logs built in so you don't have to build the images yourself to use the templating functionality.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[nnsense]

Hi there, I would ask a bit of priority for this if possible, the default fluent-bit image (using this repo as source) has only the cloudwatch_logs plugin and, in my case, redeploy the aws-for-fluent-bit just to get the old plugin would be a pain. Templating is the only way I've found to dynamically set cloudwatch logs names.

@PettitWesley since the docs are not stating the old plugin is deprecated, couldn't we temporarily have back the cloudwatch plugin, along the new one?

[gabegorelick]

couldn't we temporarily have back the cloudwatch plugin, along the new one?

You can use AWS's images: https://github.com/aws/aws-for-fluent-bit

[PettitWesley]

@PettitWesley since the docs are not stating the old plugin is deprecated, couldn't we temporarily have back the cloudwatch plugin, along the new one?

@nnsense what do you mean by "have back". The old cloudwatch plugin was never taken away 😬

[nnsense]

@PettitWesley
I mean that I cannot find it once deployed with https://github.com/fluent/helm-charts/ which is using this repo, and I suppose that should be somewhere here https://github.com/fluent/fluent-bit/tree/master/plugins, and I cannot find it, if I use cloudwatch as OUTPUT fluent-bit fails. Maybe I'm looking into the wrong place?

[PettitWesley]

@nnsense that is because the cloudwatch plugin has never been in the core of fluent bit. Its always been an external go plugin in this repo only: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit

You can only use it if you are using AWS for Fluent Bit release, the upstream (this repo) release and helm charts will not include it: https://github.com/aws/aws-for-fluent-bit

I think we have AWS for Fluent Bit helm charts here: https://github.com/aws/eks-charts

[nnsense]

Got it, thanks. I just need to wait, basically, the whole story is rather interesting: aws-for-fluent-bit chart is currently using v1.7, so no multiline core. Latest image available for that chart is still 1.8.2, which has some bugs into its multiline core implementation. That works with 1.8.3 instead, but I can't use that image with aws-for-fluent-bit without rebuilding the image and.. its plugins are not public apparently :(
So I've started using fluent-bit chart with the 1.8.3 image and life was great.. until I've find out that record accessors aren't working yet with cloudwatch_logs 😁 Yeah, I know, the sweet and sour of open source. It's fine, thanks for the explanation, I'll use aws-for-fluent-bit using the old multiline until the new 1.8.3 will be used :)

[PettitWesley]

@nnsense

That works with 1.8.3, but I can't use that image with aws-for-fluent-bit without rebuilding the image

We support 1.8.3 now:
https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.19.0

its plugins are not public apparently

What do you mean? Everything in AWS for FLuent Bit is public and open source. Nothing is closed source or hidden.

[nnsense]

Wow, that's great! When I've tried running docker build with the aws for fluent but Dockerfile I've got an authentication error at some point, I remember it was related some plugins but tomorrow I can try again and post the error, but it's not really important now that I have aws-for-fluent-bit working with 1.8.3 :)

[nnsense]

Ignore me, I've just checked and I've missed to use the makefile to build the plugins.. it has been a long day, that one :) (Also, I wasn't really happy of that path, as I should have used that custom image thereafter.. so I wasn't really convinced of what I was doing)

[julianogv]

I was able to achieve that by using

[OUTPUT]
        Name                cloudwatch
        Match               application.*
        region              ${AWS_REGION}
        log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application/$(kubernetes['namespace_name'])/$(kubernetes['labels']['app'])

[PettitWesley]

The cloudwatch plugin is our go plugin and it supports templating: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit#templating-log-group-and-stream-names

[PettitWesley]

Finally working on this: #5633