lifion stops fetching information from stream

[duinness]

This tool of yours has been behaving admirably given the load we're pushing through it, but occasionally data from a kinesis stream will stop being fetched.

The behavior where no data is fetched from a stream has been rare, but we've seen it a handful of times, and we have not been able to reproduce this behavior on demand. So far I've only seen it happen on 1 stream at a time. Others are still fetching just fine, but one will just stop even though there is more data actively being written to the stream.
Our current work-around is to cycle the containers lifion-kinesis is running on. When the tool comes back up and re-establishes the connection with kinesis it picks up where it left off.

Currently we're running 6 instances of lifion-kinesis, each in its own docker container.
Those 6 instances are monitoring ~50 kinesis streams, each stream with 1 shard.
We have also seen this happen when there is only 1 instance of lifion-kinesis.

I checked the dynamo-db created by lifion. I searched for the particular stream having an issue and found this in the 'shards' column showing that the shard is not depleted.
{ "shardId-000000000000" : { "M" : { "checkpoint" : { "S" : "49603632995838275052247271146585552269824211027765493762" }, "depleted" : { "BOOL" : false }, "leaseExpiration" : { "S" : "2020-05-22T18:30:24.553Z" }, "leaseOwner" : { "S" : "f9aV5Y9hCHisRSATtYjrGH" }, "parent" : { "NULL" : true }, "version" : { "S" : "cd6em5Roj5GzuMMa222iuB" } } }}

The 'consumers' column showed all 6 consumers with current heartbeats.

I'm not seeing clues from logging.
I thought I was really on to something with with this error recovery message, but I see the same logs even when things are working just fine.
Trying to recover from AWS.Kinesis error…\n\t- Message: Rate exceeded for stream eventing-INT-intachievewritingeventstream-31476 under account 999447569257.\n\t- Request ID: ff97d4f3-f16c-9d0f-a8d5-fec9744330b3\n\t- Code: LimitExceededException (400)\n\t- Stream: eventing-INT-intachievewritingeventstream-31476

Have you experienced this behavior in your use?

[eaviles]

Hi @duinness, the issue seems to be related to sindresorhus/got#1026 but we haven't confirmed it yet. We will take a look ASAP.

[JonFranchi]

Hi @eaviles - Has there been any progress on verifying if these are related issues?

[darrac]

Is there any movement regarding this issue?