Clarification on removal of logging templates

[romogo17]

Hi team,

I was going over some documentation associated to this repo today:

• https://aws.amazon.com/blogs/architecture/amazon-cloudwatch-insights-for-amazon-eks-on-ec2-using-aws-distro-for-opentelemetry-helm-charts
• https://github.com/aws-observability/aws-otel-helm-charts/blob/main/charts/adot-exporter-for-eks-on-ec2/documentation/metrics_to_cloudwatch_amp.md

However, I noticed that in #82, the templates for logging were removed from the helm chart, citing "stability of Logs upstream in the OTel community in 2023". I don't see any further issues or rationale behind this in the PR.

After installing the Helm chart I don't see any container logs in CloudWatch.

• Does it mean this helm chart doesn't provide the logging integration anymore and it's metrics-only?
• Or is the logging integration now supposed to be managed by the adot-collector and something in my configuration is wrong?
• Has the functionality of setting up the log collection been moved to another Helm repo? We're interested in setting this up through a Helm chart, not installing manually with manifests as this guide describes. I stumbled upon this repo by reading [EKS] [helm chart for container insights]: I would like a helm chart for installing CloudWatch Container Insights aws/containers-roadmap#779.

[romogo17]

Just found this warning in the README

Warning: Fluent Bit and Fargate logging templates are deprecated and will be removed from the Helm chart on December 30th, 2022

Given the content, why was this feature removed before the upstream feature is stabilized and available?

[mhausenblas]

@romogo17 ADOT doesn't support logs in GA yet, this was simply a cleanup to address that fact. Happy to jump on a call to discuss this is and learn more about your requirements.

[romogo17]

Thanks for the response @mhausenblas. I guess I'm just a bit confused as to why that functionality was cleaned up if the replacement with ADOT doesn't support logs in GA yet.

Sure, let's have a quick call. I'll send you an invite from my company's email address.

[mbeaucha]

Can you guys post an update of the rationale you discussed here? We too were relying on this Helm Chart to provide the fluent-bit installation as @romogo17 mentioned. Is there another chart available elsewhere for fluent-bit? The AWS Documentation still refers to this project (https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-otel.html) as providing it as well.

[mhausenblas]

@mbeaucha have a look at https://github.com/aws/eks-charts/tree/master/stable/aws-for-fluent-bit and yes, we need to update the docs everywhere as well, to clarify this.

[romogo17]

I ended up giving the aws-for-fluent-bit Helm Chart a try. Seems to be working — it's a shame it's not better documented.

CAVEATS

1. As of the day of writing, the latest chart version is 0.1.22, which uses app version 2.21.5. This version is very old and has a lot of vulnerabilities. I don't know why the Helm chart hasn't been updated in so long. I used 2.31.2 and it seems work fine.
2. Your worker nodes or K8s service account (through IRSA) will need permissions to CloudWatch. If you're using this AWS-managed policy CloudWatchAgentServerPolicy, and want to setup logRetentionDays, this policy is not enough

i-000011112222 is not authorized to perform: logs:PutRetentionPolicy on resource: arn:aws:logs:us-east-1:222333444:log-group:/aws/eks/fluentbit-cloudwatch/logs:log-stream: because no identity-based policy allows the logs:PutRetentionPolicy action

3. Typically containerinsights creates 3 log groups

   • /aws/containerinsights/cluster-name/dataplane
   • /aws/containerinsights/cluster-name/dataplane
   • /aws/containerinsights/cluster-name/dataplane

   This other helm chart only creates one: /aws/eks/fluentbit-cloudwatch/logs. This is configurable using the cloudWatch.logGroupName parameter. I don't really mind this but perhaps important to clean up the other log groups

Overall these are the values I used

    image:
      repository: public.ecr.aws/aws-observability/aws-for-fluent-bit
      tag: "2.31.2"
    cloudWatch:
      enabled: true
      logRetentionDays: 90
    firehose:
      enabled: false
    kinesis:
      enabled: false
    elasticsearch:
      enabled: false

[mhausenblas]

@romogo17 thanks a bunch for that and as discussed would be great if you wanted to PR the README with a sentence to point to this (feel free to link to this issue here for "more details")?