For REST requests, remember successful fallback hosts for 10 minutes

content/client-lib-development-guide/features.textile

@@ -87,9 +87,10 @@ h3(#restclient). RestClient
 * @(RSC13)@ The client library must use the connection and request timeouts specified in the @ClientOptions@, falling back to the defaults described in @ClientOptions@ below
 * @(RSC15)@ Host Fallback
 ** @(RSC15b)@ The fallback behavior described below only applies when the default @rest.ably.io@ endpoint is being used and has not been overriden (see "RSC11":#RSC11), @ClientOptions#fallbackHostsUseDefault@ is @true@, or an array of @ClientOptions#fallbackHosts@ is provided. If host fallback is not supported, failing HTTP requests that would have "qualified for a retry against a fallback host (see RSC15d)":#RSC15d, will instead result in an error immediately
-** @(RSC15e)@ By default, every new HTTP request is first attempted to the default primary host @rest.ably.io@ (unless overriden in @ClientOptions#restHost@), which, through DNS, is automatically routed to the client's closest data center. The client library must always prefer the default endpoint (closest data center), even if a previous request to that endpoint has failed
+** @(RSC15e)@ The primary host is by default @rest.ably.io@ (unless overriden in @ClientOptions#environment@ or @ClientOptions#restHost@), which, through DNS, is automatically routed to the client's closest data center. New HTTP requests (except where @RSC15f@ applies and a cached fallback host is in effect) are first attempted against the primary host.
 ** @(RSC15a)@ In the case of an error necessitating use of an alternative host (see "RSC15d":#RSC15d), try fallback hosts (with a matching Host header as this is necessary when fallbacks are proxied through a CDN) in random order, continuing to try further hosts if "qualifying errors":#RSC15d occur, failing when all have been tried or the configured @httpMaxRetryCount@ has been reached (see "TO3l@":#TO3l5). This ensures that a client library is able to work around routing or other problems for the user's closest data center. For example, if a @POST@ request to @rest.ably.io@ fails because the default endpoint is unreachable or unserviceable, then the @POST@ request should be retried again against the fallback hosts in attempt to find an alternate healthy data center to service the request. The five default fallback hosts are @[a-e].ably-realtime.com@. If an array of custom fallback hosts are provided in @ClientOptions#fallbackHosts@, then they will be used instead. If an empty array of fallback hosts is provided, then fallback host functionality is disabled
 ** @(RSC15d)@ Errors that necessitate use of an alternative host include: host unresolvable or unreachable, request timeout, or a response but with an applicable HTTP status code in the range @500 <= code <= 504@. Resending requests that have failed for other failure conditions will not fix the problem and will simply increase the load on other data-centers unnecessarily
+** @(RSC15f)@ Once/if a given fallback host succeeds, the client should store that successful fallback host for @ClientOptions.fallbackRetryTimeout@. Future HTTP requests during that period should use that host. If during this period a "qualifying errors":#RSC15d occurs on that host, or after @fallbackRetryTimeout@ has expired, it should be unstored, and the fallback sequence begun again from scratch, starting with the default primary host (@rest.ably.io@ or @ClientOptions#restHost@)
 * @(RSC17)@ When instancing the library, if a @clientId@ attribute is set in @ClientOptions@, then the @Auth#clientId@ attribute will contain the provided @clientId@
 * @(RSC19)@ @RestClient#request@ function is provided as a convenience for customers who wish to use bleeding edge REST API functionality that is either not documented or is not included in the API for our client libraries. The REST client library provides a function to issue HTTP requests to the Ably endpoints with all the built in functionality of the library such as authentication, paging, fallback hosts, MsgPack and JSON support etc. The function:
 ** @(RSC19a)@ Method signature is @request(string method, string path, Dict<String, String> params?, JsonObject | JsonArray body?, Dict<String, String> headers?) -> HttpPaginatedResponse@ with arguments: @method@ is a valid "HTTP verb":https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html (must support @"GET"@, @"POST"@, and @"PUT"@, should support @"PATCH"@ and @"DELETE", may support others); @path@ is the path component of the URL such as @"/channels"@; @params@ and @headers@ are optional arguments containing pairs of key value strings (multi-valued headers are not supported) that will result in query params and HTTP headers being added respectively in the request (the argument types can be idiomatic for the language such as @Object@ in the case of Javascript); @body@ is an optional @JsonObject@ or @JsonArray@ like object argument that can be easily serialized to MsgPack or JSON
@@ -429,7 +430,7 @@ h3(#realtime-connection). Connection
 ** @(RTN16f)@ The @msgSerial@ component of the @recoveryKey@, unlike the other two components, is not sent to Ably, but rather is used to set the library internal @msgSerial@. (If the recover fails, the counter should be reset to 0 per "RTN15c3":#RTN15c3 )
 * @(RTN17)@ Host Fallback
 ** @(RTN17b)@ The fallback behavior described below only applies when the default @realtime.ably.io@ endpoint is being used and has not been overriden (see "RTC1d":#RTC1d and "RTC1e":#RTC1e), @ClientOptions#fallbackHostsUseDefault@ is @true@, or an array of @ClientOptions#fallbackHosts@ is provided.
-** @(RTN17a)@ By default, every connection attempt is first attempted to the default primary host @realtime.ably.io@ (unless overriden in @ClientOptions#realtimeHost@), which, through DNS, is automatically routed to the client's closest data center. The client library must always prefer the default endpoint (closest data center), even if a previous connection attempt to that endpoint has failed
+** @(RTN17a)@ By default, every connection attempt is first attempted to the default primary host @realtime.ably.io@ (unless overriden in @ClientOptions#realtimeHost@), which, through DNS, is automatically routed to the client's closest data center. The client library must always prefer the default endpoint (closest data center), even if a previous connection attempt to that endpoint has failed. (That is, @RSC15f@ does not apply)
 ** @(RTN17c)@ In the case of an error necessitating use of an alternative host (see "RTN17d":#RTN17d), the @Connection@ manager should first check if an internet connection is available by issuing a @GET@ request to @https://internet-up.ably-realtime.com/is-the-internet-up.txt@. If the request succeeds and the text "yes" is included in the body, then the client library can assume it has a viable internet connection and should then immediately retry the connection against all fallback hosts to find an alternative healthy data center. The five default fallback hosts are @[a-e].ably-realtime.com@ and should be attempted in random order. The connection requests must include a matching Host header as this is necessary when fallbacks are proxied through a CDN. See "RSC15a":#RSC15a for details on how custom fallback hosts are specified and used
 ** @(RTN17d)@ Errors that necessitate use of an alternative host include: host unresolvable or unreachable, connection timeout, or a response but with an "error body @statusCode@":/rest-api#error-response or HTTP response status code in the range @500 <= code <= 504@. Attempting to reconnect to a fallback host for other failure conditions will not fix the problem and will simply increase the load on other data-centers unnecessarily
 ** @(RTN17e)@ If the realtime client is connected to a fallback host endpoint, then for the duration that the transport is connected to that host, all HTTP requests, such as history or token requests, should be first attempted to the same data center the realtime connection is established with i.e. the same fallback host must be used as the default HTTP request host. If however the HTTP request against that fallback host fails, then the normal fallback host behavior should be followed attempting the request against another fallback host as described in "RSC15":#RSC15
@@ -1290,6 +1291,7 @@ h4. ClientOptions
 **** @(TO3l8d)@ The size of the @extras@ property is the string length of its JSON representation
 **** @(TO3l8e)@ The size of a @null@ or omitted property is zero
 *** @(TO3l9)@ @maxFrameSize@ integer - default 524288 (512KiB). The maximum size of a single POST body or "WebSocket":/concepts/websockets frame. This is mostly only relevant for `Rest#request` (e.g. for batch publishes), since publishes will hit the @maxMessageSize@ limit before this
+*** @(TO3l10)@ @fallbackRetryTimeout@ integer - default 600000 (10 minutes). (After a failed request to the default endpoint, followed by a successful request to a fallback endpoint), the period in milliseconds before HTTP requests are retried  against the default endpoint
 
 h4(#token-params). TokenParams
 * @(TK1)@ A class providing parameters of a token request. These params are used when invoking @Auth#authorize@, @Auth#requestToken@ and @Auth#createTokenRequest@