fix: Throttle AppSync LimitExceeded errors

[lawmicha]

Issue #, if available:

Description of changes:
The LimitExceeded error is received when AppSync throttles the client for making too many subscription requests:

{"type":"error","payload":{"errors":{"errorType":"LimitExceededError","message":"Rate limit exceeded"}}}

This happens once the rate of subscribes has exceeded the limit on AppSync. Since 100 subscriptions can be established successfully, a loop with 200 or more subscriptions requests will reproduce this.

Consider the current flow

flowchart
A[AppSync] --> |sends LimitExceeded| B[Websocket client];
B --> |convert to ConnectionProviderError.other| C1[Subscription1 - subscribed];
B --> |convert to ConnectionProviderError.other| C2[Subscription2 - not subscribed];
C1 --> |resultHandler error| D[Caller];
C2 --> |resultHandler error| D;

Loading

1. Subscribe 200 times, 100 will be successfully connected. 101th and onward would get MaxSubscriptionsReachedError(subscriptionId) (and go into retry logic). But, when sending 101st to 200th subscription requests all at the same time, AppSync throttles this client with error LimitExceededError.
2. LimitExceededError is received, translate this to a generic ConnectionProviderError.other because there's no id in the response object. Send ConnectionProviderError.other to all 200 subscriptions (even the subscribed ones).
3. All subscriptions will transition to .notSubscribed and return a failed event to the caller. Let's say 100 out of 200 subscriptions are throttled, then all 200 subscriptions will get a callback for each error, which is 20000 callbacks. Even though we removed the listener from the connection provider in the previous PR fix: Subscription failed event should be terminal event #74 , the problem is that connection makes a copy of the subscription listeners before broadcasting the message, so even though they are being removed after sending the error from a previous subscription, they continue to emit a failed message. In normal message rates, removing the connection is enough to do the job.

How should we handle LimitExceededError?

1. The connection receives a LimitExceededError without id (w/o subscription id - we'll treat this as a connection level LimitExceededError.). Convert it over to .limitExceeded(nil) error.
2. Pass it to a client side throttle to slow down the rate at which subscriptions will receive this error.
3. Throttle for 150 ms, and then pass it to the subscriptions.
4. A subscribed subscripiton will ignore this error.
5. An in progress subscription will return the failure.

flowchart
A[AppSync] --> |sends LimitExceeded| B[Websocket client];
B --> |convert to ConnectionProviderError.limitExceeded w/ nil| C[Throttle];
C --> |send throttled limitExceeded error| C1[Subscription1 - subscribed];
C --> |send throttled limitExceeded error| C2[Subscription2 - notSubscribed];
C1 --> D[no-op];
C2 --> |resultHandler error| E[Caller];

Loading

Manual testing

When subscribing to 500 subscriptions, 100 are connected successfully, 400 failed events. Send a mutation, 100 subscription events are received.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[diegocstn]

nit: it might be worth to extract all these error types (LimitExceededError, MaxSubscriptionsReachedError, MaxSubscriptionsReachedException) in an enum

[lawmicha]

I can do this in the next PR -> https://github.com/aws-amplify/aws-appsync-realtime-client-ios/pull/69/files it is also error handling related