approach to retries

[harrysolovay]

When an RPC server disconnects unexpectedly, do we attempt to reconnect, re-send any egress messages, misc? How about for transactions? If the user has encoded the transaction with a nonce and window of validity, yes. What if they have not? Should users need to explicitly opt into retry behavior? Should they specify their own back-off algorithm? Should they specify their own fallback logic beyond simply re-sending? A lot of questions to answer regarding approach to retries.

[kratico]

Message retry seems to depend on the message type or the operation that will be triggered by that message.

Perhaps messages can be categorized in idempotent vs non-idempotent messages (like REST) depending on the operation that they trigger.

For example,

• in a subscription to chain_subscribeNewHead, a subscription retry should be fine
• in a author_submitAndWatchExtrinsic, an extrinsic retry should not be fine but a retry to monitor the extrinsic events should be fine

This leads me to think that the message producer needs some context to decide if a call/subscription can be retried.

Then, for message delivery strategies, there could be messages that are

• fire and forget
• fire and acknowledge success/failure
• fire, retry x times and acknowledge success/failure
• fire, timeout and ...

[harrysolovay]

I completely agree with everything you wrote. Especially...

Perhaps messages can be categorized in idempotent vs non-idempotent messages

... and...

the message producer needs some context to decide if a call/subscription can be retried.

I think it's up to us to ensure that the necessary context is accessible to the developer, who can use it alongside their own monitoring/resubmission logic. Mocking out some example scenarios would be a good idea... assuming we think message-specific retries is an early priority (not so sure that it is––what're your thoughts?).

However... for fun, let's play this out a bit more. Your 4 bullet points highlight some key considerations, such as how do we identify whether a non-idempotent message has been received?, what stage it is in?, do we need to await confirmation that the message did not go through before resubmission?.

As I write out these questions, the hard-coding of use-case specific handling seems undesirable––even for std effects. Even if we do it "right," it will be "wrong" for some consumers. Probably best to steer clear of most message-related retries for now.

[kratico]

As I write out these questions, the hard-coding of use-case specific handling seems undesirable––even for std effects. Even if we do it "right," it will be "wrong" for some consumers. Probably best to steer clear of most message-related retries for now.

Yes, it seems that message retry should not be responsibility of the RPC client.
For now, it would be better to throw an error (as we currently do) and let the developer do the resubmission depending on the use-case.
In the future, we could add some helper utils to do the retries and I believe that those helpers do not need to be part of the RPC client implementation.

[tjjfvi]

For reference: #492 #502

[harrysolovay]

Let's create an examples/retries directory and showcase common things that one would want to retry. For instance, a redacted.eg.ts.