Container Insights unit mismatch

[corey-cole]

Describe the bug
The CPU utilization and reservation metrics units don't match the official Container Insights units.

Steps to reproduce

1. Enable Container Insights on an ECS Fargate cluster
2. Launch a workload into the cluster
3. Observe the CpuReserved and CpuUtilized metrics
4. Disable Container Insights
5. Launch an OpenTelemetry sidecar using configuration from here: https://aws-otel.github.io/docs/components/ecs-metrics-receiver
6. Observe CpuReserved and CpuUtilized metrics from OpenTelemetry exporter. Confirm units via EMF stored in CloudWatch logs.
7. Observe change in units.

What did you expect to see?
I expected to see that the units would be the same for official ECS Container Insights and the sidecar version.

What did you see instead?
CPU reservation was given as percentage of a vCPU (e.g. 0.256) instead of CPU shares (256)

What version did you use?
Version: v0.15.1

What config did you use?
Config: The config file given on this page (https://aws-otel.github.io/docs/components/ecs-metrics-receiver) in the subsection "Full configuration for task level metrics"

Environment
OS: Whatever the OS is for amazon/aws-otel-collector:latest with sha256:713953f93619702dee9000465f430d8179a6b9356011559159312979f18c513f
Compiler(if manually compiled): N/A

Additional context
Add any other context about the problem here.

[jpkrohling]

@anuraaga, could you take a look as the code owner?

[sethAmazon]

@alolita can I take up this issue?

[sethAmazon]

I have tested using the steps above and confirmed the bug will follow up with ecs team.

[hossain-rayhan]

I believe there is slight difference between CpuReservation vs CpuReserved. Same for CpuUtilization vs CpuUtilized. This were discussed with ECS Insights team member Chi for clarification.

In the Container Insights dashboard, I believe we only show reservede and utilized metrics, and the unit should be Percentage and vCpu. Can you pull the reference with which you are matching?

[gavindoudou]

When using ECS container insights, I checked CPU Utilization and Memory Utilization metrics, the unit for them is Percent. It matches when using the sidecar solution: the doc here https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awsecscontainermetricsreceiver: ecs.task.cpu.utilized | container.cpu.utilized | Percent. Can you be more specific about what in ECS container insights and what in side-car solution are you comparing?

[gavindoudou]

Talked with Seth, are trying to reproduce the sidecar solution.

[gavindoudou]

Reproduced the issue. This is a bug. The CPU utilized metric is different (around 100 times) when using ECS container insights and Sidecar solution. I spoke with Rayhan and we agreed to change on our side (ADOT) rather than on the CW console side. I will investigate further to see how can we fix that.

[sethAmazon]

@corey-cole can you please post your config. I'm using the task cpu reserved and utilization as per the sample config in the aws otel collector. I want to know if you are using the same.

[sethAmazon]

https://github.com/aws-observability/aws-otel-collector/blob/main/config/ecs/container-insights/otel-task-metrics-config.yaml is what I'm using

[codeboten]

Closing as this has been waiting for author since Jul 5

[bmbferreira]

can we reopen this? I still see some differences between the metrics coming from awsecscontainermetrics receiver and the ones I see on the aws container insights.