Handle "FatalError" PUSH errors

[vladsud]

New class of disconnection errors exists that we are not taking proper care - shows up as "Disconnect: Error: FatalError" reason for fluid:telemetry:Container:ConnectionStateChange_Disconnected

Per Gary, this happens when PUSH can't persist ops to storage.
While this is likely broad category, right now all issues like that happened due to SPO losing ops.
So the right approach is

1. Make this critical error in SPO driver (flip canRetry to true where we treat "server_disconnect")
2. Add new type of error (i.e. make it typed - see IError & specific types like IThrottlingError), and work with Scriptor to handle it properly.

@GaryWilber - FYI

[vladsud]

@GaryWilber , is there any way to have unique code for that issue? I think when I looked at telemetry I did not see any, but I might be wrong. Currently, we do not make decisions based on strings, only based on error code, it would be great to keep that.

[GaryWilber]

The code we return for FatalError is 500. You should use that to make decisions.

[vladsud]

Is this the only cause for 500? i.e. does 500 always mean FatalError?
Wes - it might be worth to build a "validation" layer, where we have both error code & string and validate (telemetry only) that our expectations are matching, but use status code for all decision making

[GaryWilber]

Yea it's the only cause for 500.