RetriableDocumentStorageService is not handling retries correctly

[vladsud]

APIs know to be broken:

• RetriableDocumentStorageService.write
• RetriableDocumentStorageService.getVersion
• RetriableDocumentStorageService.writeSummaryTree

In particular, we do not handle 429s correctly, or any other retryable error.

Can we move away from using DocumentStorageServiceProxy here? It hides details, it's not obvious on initial review what APIs are wrapped vs. not.
As other new APIs would get added, developers would keep missing the need to change that class.

Note that while it's right place to solve these issues, it creates another problem - too much noise in telemetry when we are offline. That's because actual driver calls will generate events, and if we call them every 8 seconds, they will generate a ton of data for couple hours of being offline (all of that data is cached and flushed to Aria on connection).
We likely need to create a separate issue to think about solution to this problem.
For example, op fetching does not log anything in the driver, DeltaManager is responsible for all logging. Maybe we need to adopt similar thing and say that RetriableDocumentStorageService is doing all loging?

[vladsud]

Also we need to ensure there is telemetry in place (number of attempts, total duration, if possible - last error we deal with).
I do not remember if we have dedicated issue to just that. If not, would be great to take care of it here.

[jatgarg]

Moving the telemetry part to other issue: #4855