API call for Kinesis HealthCheck: Structure

[danotorrey]

Overall structure for the Kinesis HealthCheck (resource endpoint, tie together log retrieval, detection, and parsing).

Three other issues exist as part of this larger issue:

• Retrieval: API call for Kinesis HealthCheck: Log retrieval #58
• Detection: API call for Kinesis HealthCheck: Detection #60
• Parsing: API call for Kinesis HealthCheck: Parsing #59

More info:
Note: Before this can be implemented, we first need to identify if we can read a message from a Kinesis stream without deleting it from the stream (to avoid it being lost). The Kinesis Client Library for Java will create a consumer that retrieves a large number of message at a time and does not support the ability to leave a message on a stream. We need to do some research to identify if a workaround exists to read a sample message from the stream and leave it in the stream.

See https://docs.aws.amazon.com/streams/latest/dev/kcl-migration.html for the new KCL config options.

An issue was opened with this Kinesis Client Library team, and they recommended using the direct kinesis API.

Implement the HealthCheck API call. This should perform the following operations:

Attempt to pull a log message from a Kinesis stream see Kinesis consumer code in this issue, also see this sample consumer/subscriber code for how to use the pattern in the latest Kinesis client library.

Identify if the log message is in a known format. Indicate that format in the response message. This will automatically establish a codec/parsing for it. Perhaps we can use a regex for this? See this link that lists the format for all AWS services that can publish messages to CloudWatch: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html We definitely have to support CloudWatch and supporting others would be a plus.

• Return log message for parsing.

[danotorrey]

Closed via #86