Adding a new metric: Application-level MillisBehindLatest

[QAQJ]

Feature request discussion: #249

According the document, MillisBehindLatest metric is used for monitoring how far the KCL application is behind to the latest record in the shard it's working with, and this is a shard-level metric. However a KCL application may work with multiple shards (or even streams), and as per customer's request, an application-level MillisBehindLatest metric would be helpful so that they only need to setup 1 alarm for monitoring the latencies of their KCL application, instead of setting an alarm for each shard.

This PR implements the application-level MillisBehindLatest metric, simply by creating a new application-level metrics scope, along with the existing shard-level metrics scope, in ProcessTask, and use it to collect and publish the MillisBehindLatest data. This scope doesn't have ShardId or StreamId as its dimensions, so all the MillisBehindLatest will be published to a general metric in CloudWatch.

To view this metric, customer can log into their AWS console -> CloudWatch -> All Metrics -> name-of-their-application under Custom Namespaces section -> Operation -> then they should see the metric like below.

The metric level is set to SUMMARY, same as shard-level MillisBehindLatest metric.

[avahuang0429]

Naming: Read through the scope of ProcessTask.java and MetricsUtil.java. For consistency, let's keep the naming *_METRIC reserved for data fields. What we want to achieve here is a different scope hence it cannot be categorized as *_METRIC. You could either rename to operation or scope.
Suggestion: either use OPERATION or SCOPE

[avahuang0429]

I read the public documentation and found "ApplicationLevelMetrics" not descriptive enough. This new metric we are adding belongs to https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html#kcl-metrics-per-app. Refer to this naming convention on Operation field, and get feedback from PM on the naming.

[QAQJ]

I showed a demo to the PM previously, but yeah I agree, will talk with the PM to confirm the names.

[QAQJ]

After talking with Nihar, we decided to use ApplicationTracker as the operation, and the code is updated now.

[avahuang0429]

General comment, can we try test the correctness of the metrics? I tried to look for test cases but there's none. Setting up brand new test case could be an option, but evaluate either that or manual testing

[QAQJ]

@avahuang0429 I do have a testing script in my personal repo: https://github.com/QAQJ/amazon-kinesis-client/blob/dev-jiqilin/amazon-kinesis-client/src/test/java/software/amazon/kinesis/ApplicationTest.java, which runs a 2-stream application and the metric can be published to the cloudwatch. However the concern is that I wasn't able to create any latency to that application, so all MillsBehindLatest (including all shards & app-level) are all 0 my test. But theoretically, if any latency happens in any shard, by go to Graphed Metrics and choose "Maximum" in the Statistic dimension, like the second screenshot, customer should be able to see it.


o

[avahuang0429]

LGTM, make sure to paste offline comments + retry build till pass before merging

[QAQJ]

In my personal branch I've manually created some MillisBehindLatest latency: https://github.com/QAQJ/amazon-kinesis-client/tree/dev-jiqilin
When we set the Statistics to Average the metrics will show as the first image, we can see the ApplicationTracker.MillisBehindLatest is the average of the 2 shards.

When we set the Statistics to Maximum, ApplicationTracker.MillisBehindLatest will be the same as the shard with largest latency.

Thus, KCL users can decide which statistic they want for their alarms.