[FEATURE REQ]Immediately trigger cross-regional fail-over on 410/LeaseNotFound(1022) and 429/SystemResourceUnavailable(3092)

[FabianMeiswinkel]

Is your feature request related to a problem? Please describe.
Today when a read region loses the read lease it returns 410/LeaseNotFound - the client will then retry locally for up-to 30 seconds (catch-all 410 retry like for 410/0 or connectivity related 410) .

Describe the solution you'd like

• Instead the 410/LeaseNotFound(1022) should be mapped immediately to a 503 to allow the cross-regional retry to happen as quickly as possible (it should also result in per partition mark-down when available) - retrying in the next preferred region like usual for reads is acceptable.

• For 429/3092 (Request Throttled/ SystemResourceUnavailable), we would only want to short-circuit this to retry cross-region quickly when we can actually retry cross region (not a write in single master) - and for reads only when it is returned from all replica in a region. The latter is extremely unlikely - so, I would probably compromise on a pragmatic approach and simply short-circuit by mapping 429/3092 to a 503 for write operations when multi-master is enabled (and there is >1 write regions).
  That way we get the cross-region retry in the scenario where this is most critical.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Description Added
• Expected solution specified