Metric Value List can go above MetricDefinition limits, causing CW missing metrics

[axeliux]

Apparently if you call MetricsLogger.putMetric() more than 100 times, it will keep adding values to the metric causing to reach out of it's limits. This would cause missing metrics as the log entry would not be processed.

Reference:

Metrics— An array of MetricDefinition objects. This array MUST NOT contain more than 100 MetricDefinition objects.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html

[axeliux]

Maybe we are not limiting the size of the metric value at all.

aws-embedded-metrics-java/src/main/java/software/amazon/cloudwatchlogs/emf/model/MetricDirective.java

Line 64 in b6a99c0

metrics.get(key).addValue(value);

[msailes]

It's not just the number of metrics, it's all validation.

Size of numbers, infinity...

[axeliux]

I wonder if instead of having a new ArrayList<> to store the metric values, we could have something like ```EvictingQueue`` of a fixed size 100.

I understand that we could lose some metrics values, but given the current 100 limitation, it would be better than not having a metric at all.

aws-embedded-metrics-java/src/main/java/software/amazon/cloudwatchlogs/emf/model/MetricDefinition.java

Line 49 in b6a99c0

this(name, Unit.NONE, new ArrayList<>());

https://guava.dev/releases/snapshot/api/docs/com/google/common/collect/EvictingQueue.html

[axeliux]

Not quite the same issue, but this handle a limit on the number of metrics we can process per log:

https://github.com/awslabs/aws-embedded-metrics-java/pull/42/files/dc47697d0a860c46dbbc06669558452ffa28e117

[jaredcnance]

This problem will likely be handled in a similar way as the one you linked. We will continue to accept additional metrics and at serialization time, split it across multiple events. You can see how this was implemented for python. Regarding @msailes's comment, yes we would like to add improved validation to ensure we only accept valid dimensions, metric names, namespaces, and metric values. Eventually, we'd also like to support client-side aggregation which would allow you to include StatisticSets and Values/Counts similar to MetricDatums.