Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose MillisBehindLatest from GetRecords() call #27

Closed
wushujames opened this issue Jun 16, 2015 · 6 comments
Closed

Expose MillisBehindLatest from GetRecords() call #27

wushujames opened this issue Jun 16, 2015 · 6 comments

Comments

@wushujames
Copy link

The Kinesis API recently started returning MillisBehindLatest in the response to GetRecords().

http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html

MillisBehindLatest
The number of milliseconds the GetRecords response is from the tip of the stream, indicating how far behind current time the consumer is. A value of zero indicates record processing is caught up, and there are no new records to process at this moment.

Type: Long

Valid range: Minimum value of 0.

The current KCL only exposes the records to the IRecordProcessor. It would be useful to expose the MillisBehindLatest to the IRecordProcessors as well, so they would know how far behind they are.

@dosani
Copy link
Contributor

dosani commented Jun 16, 2015

Hi @wushujames,

Thanks for the suggestion. We are considering adding this feature. Would you be able to share how you are planning to use this metric in your application?

Btw, in case you weren't aware, KCL currently sends this metric to Cloudwatch (which may be useful for monitoring purposes).

Thanks,
-Adnan.

@jqr
Copy link

jqr commented Jun 16, 2015

I am also interested in this feature, particularly when exposed to the MultiLangDaemon protocol. My use-case is easy enough to explain:

We batch thousands of smaller subrecords into each Kinesis record and we track latency of these subrecords individually, but to track the record itself we're presently using the ExplicitHashKey to the same thing that HashKey is normally set to, but this allows us to use HashKey to store the time of record insertion and compute latency of Kinesis itself while not changing any Kinesis behavior.

This little bit of clever work-around gives us a per-record metadata field and the ability to easily use our own monitoring and alerting systems with extremely low latency. Cloudwatch is more difficult and less effective in this case due to reporting latency and having to query another system for simple data that is actually provided by the underlying API.

There's a way to address most of this in our record encoding format, to add this data in, but by adding it within the Kinesis record we would be required to do another layer of encoding and encapsulation for 1 simple field.

@wushujames
Copy link
Author

I only just recently discovered that it is in CloudWatch. Thanks for that.

We have an application that needs to be mostly caught up to "now" before it is allowed to answer API requests. 1 minute back from "now" might be acceptable, 5 minutes might be acceptable, but hours back is unacceptable. This metric would let the application and operations decide when it is "mostly caught up" and when the application is allowed to start servicing requests.

Since this metric is per GetRecords() call, and the GetRecords() calls is per shard, we would have to have some heuristics that examines the MillisBehindLatest value for each shard, and only takes action when all shards have MillisBehindLatest lower than our desired threshold.

@stephenh
Copy link

stephenh commented Jul 9, 2015

I was looking to get MillisBehindLatest in IRecordProcessor as well, because we want to report it to CW.

The KCL already reports it to CW, but AFAICT it's only reported on the (combined) Operation+ShardId dimension, which means we don't have a single application-wide metric/dimension that we could use for both CW dashboards and auto scaling, e.g. to scale/up down based on MillisBehindLatest.

(Or if I want to put MillisBehindLatest in a CW dashboard, and I've got ~50 shards, I've got to select all 50 lines one-by-one. And then reselect 100-shards if I reshard from to 50 to 100.)

Is there an easy way for KCL to publish MillisBehindLatest both with and without shard id? Or am I overlooking something in CW?

@stephenh
Copy link

stephenh commented Jul 9, 2015

You have that nice ProcessRecordsInput now, it wouldn't even be a breaking change. :-)

stephenh added a commit to stephenh/amazon-kinesis-client that referenced this issue Jul 9, 2015
@manango
Copy link
Contributor

manango commented Jul 9, 2015

The KCL exposes MillisBehindLatest value to the record processor in 1.5.0 release. Thanks everyone for feedback.

@manango manango closed this as completed Jul 9, 2015
ychunxue pushed a commit to ychunxue/amazon-kinesis-client that referenced this issue Jun 17, 2020
ychunxue pushed a commit to ychunxue/amazon-kinesis-client that referenced this issue Jun 17, 2020
[Cherry-pick from multi_stream_2] Empty lease response for ResourceNotFound exceptions (awslabs#27)
zengyu714 added a commit that referenced this issue Dec 22, 2021
* Configure dependabot

* Bump guava from 26.0-jre to 31.0.1-jre (#26)

Bumps [guava](https://github.com/google/guava) from 26.0-jre to 31.0.1-jre.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)

---
updated-dependencies:
- dependency-name: com.google.guava:guava
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump junit from 4.11 to 4.13.2 (#28)

Bumps [junit](https://github.com/junit-team/junit4) from 4.11 to 4.13.2.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.11.md)
- [Commits](junit-team/junit4@r4.11...r4.13.2)

---
updated-dependencies:
- dependency-name: junit:junit
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump aws-java-sdk.version from 1.12.128 to 1.12.130 (#27)

Bumps `aws-java-sdk.version` from 1.12.128 to 1.12.130.

Updates `aws-java-sdk-dynamodb` from 1.12.128 to 1.12.130
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](aws/aws-sdk-java@1.12.128...1.12.130)

Updates `aws-java-sdk-kinesis` from 1.12.128 to 1.12.130
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](aws/aws-sdk-java@1.12.128...1.12.130)

Updates `aws-java-sdk-cloudwatch` from 1.12.128 to 1.12.130
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](aws/aws-sdk-java@1.12.128...1.12.130)

---
updated-dependencies:
- dependency-name: com.amazonaws:aws-java-sdk-dynamodb
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.amazonaws:aws-java-sdk-kinesis
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.amazonaws:aws-java-sdk-cloudwatch
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump protobuf-java from 3.11.4 to 3.19.1 (#25)

Bumps [protobuf-java](https://github.com/protocolbuffers/protobuf) from 3.11.4 to 3.19.1.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/master/generate_changelog.py)
- [Commits](protocolbuffers/protobuf@v3.11.4...v3.19.1)

---
updated-dependencies:
- dependency-name: com.google.protobuf:protobuf-java
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump maven-failsafe-plugin from 2.19.1 to 2.22.2 (#23)

Bumps [maven-failsafe-plugin](https://github.com/apache/maven-surefire) from 2.19.1 to 2.22.2.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](apache/maven-surefire@surefire-2.19.1...surefire-2.22.2)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-failsafe-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Remove dependabot configuration in v1.x branch as it's been configured in master

* Bump maven-gpg-plugin from 1.6 to 3.0.1 (#33)

Bumps [maven-gpg-plugin](https://github.com/apache/maven-gpg-plugin) from 1.6 to 3.0.1.
- [Release notes](https://github.com/apache/maven-gpg-plugin/releases)
- [Commits](apache/maven-gpg-plugin@maven-gpg-plugin-1.6...maven-gpg-plugin-3.0.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-gpg-plugin
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump maven-compiler-plugin from 3.2 to 3.8.1 (#32)

Bumps [maven-compiler-plugin](https://github.com/apache/maven-compiler-plugin) from 3.2 to 3.8.1.
- [Release notes](https://github.com/apache/maven-compiler-plugin/releases)
- [Commits](apache/maven-compiler-plugin@maven-compiler-plugin-3.2...maven-compiler-plugin-3.8.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-compiler-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump lombok from 1.16.10 to 1.18.22 (#31)

Bumps [lombok](https://github.com/projectlombok/lombok) from 1.16.10 to 1.18.22.
- [Release notes](https://github.com/projectlombok/lombok/releases)
- [Changelog](https://github.com/projectlombok/lombok/blob/master/doc/changelog.markdown)
- [Commits](projectlombok/lombok@v1.16.10...v1.18.22)

---
updated-dependencies:
- dependency-name: org.projectlombok:lombok
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump maven-javadoc-plugin from 2.10.3 to 3.3.1 (#30)

Bumps [maven-javadoc-plugin](https://github.com/apache/maven-javadoc-plugin) from 2.10.3 to 3.3.1.
- [Release notes](https://github.com/apache/maven-javadoc-plugin/releases)
- [Commits](apache/maven-javadoc-plugin@maven-javadoc-plugin-2.10.3...maven-javadoc-plugin-3.3.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-javadoc-plugin
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump maven-surefire-plugin from 2.19.1 to 2.22.2 (#29)

Bumps [maven-surefire-plugin](https://github.com/apache/maven-surefire) from 2.19.1 to 2.22.2.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](apache/maven-surefire@surefire-2.19.1...surefire-2.22.2)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-surefire-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump log4j-core from 2.16.0 to 2.17.0 (#36)

Bumps log4j-core from 2.16.0 to 2.17.0.

---
updated-dependencies:
- dependency-name: org.apache.logging.log4j:log4j-core
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump commons-lang3 from 3.7 to 3.12.0 (#38)

Bumps commons-lang3 from 3.7 to 3.12.0.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-lang3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump DynamoDBLocal from 1.11.86 to 1.17.2 (#37)

Bumps DynamoDBLocal from 1.11.86 to 1.17.2.

---
updated-dependencies:
- dependency-name: com.amazonaws:DynamoDBLocal
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump maven-source-plugin from 3.0.1 to 3.2.1 (#35)

Bumps [maven-source-plugin](https://github.com/apache/maven-source-plugin) from 3.0.1 to 3.2.1.
- [Release notes](https://github.com/apache/maven-source-plugin/releases)
- [Commits](apache/maven-source-plugin@maven-source-plugin-3.0.1...maven-source-plugin-3.2.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-source-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump commons-logging from 1.1.3 to 1.2 (#34)

Bumps commons-logging from 1.1.3 to 1.2.

---
updated-dependencies:
- dependency-name: commons-logging:commons-logging
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add test dependencies

Co-authored-by: Yu Zeng <yuzen@amazon.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants