Implementation for resolving data corruption issue for distributed data ordering service

[jatgarg]

Fixes: #4399
Sol:
Whenever we receive a disconnected event, if we have outstanding ops for disconnected client and if that client was write client and we are not already waiting, then we start a timer for waiting for leave/timeout of that client. If we receive the addMember/join of new client before receiving removeMember/leave of older client, then we don't move container to connected state instead we wait for leave/timeout. So later when leave come, we move the container to Connected state. If leave of older client is received first, then we just move container to connected state on join immediately. In case of timeout, we stop waiting for leave and just continue with normal operation hoping that nothing bad happens.
Current wait time for leave op is 90 sec.

[msfluid-bot]

⯅ @fluidframework/base-host: +2.38 KB

Metric Name	Baseline Size	Compare Size	Size Diff
main.js	179.99 KB	182.37 KB	⯅ +2.38 KB
Total Size	179.99 KB	182.37 KB	⯅ +2.38 KB

■ @fluid-example/bundle-size-tests: No change

Metric Name	Baseline Size	Compare Size	Size Diff
container.js	201.54 KB	201.54 KB	■ No change
map.js	49.11 KB	49.11 KB	■ No change
matrix.js	143.42 KB	143.42 KB	■ No change
odspDriver.js	201.22 KB	201.22 KB	■ No change
sharedString.js	158.14 KB	158.14 KB	■ No change
Total Size	753.45 KB	753.45 KB	■ No change

---

Baseline commit: 0b4889b

Generated by 🚫 dangerJS against 3c76d63

[anthony-murphy]

general feedback. can we encapsulat the logic we are adding to container into a class. i don't like the way the logic and state is so spread out. the class should also then be unit testable, so fewer end to end tests will be needed, consider then when builing the api

[vladsud]

I personally do not see much value in waiting 5 min - it always same as server timeout (server has 5 min timeout, the difference is only in TCP/IP timeout that adds on top of it for worse possible case).
I'm of an opinion that we want to go with much smaller timeout as we are not making things worse then they are today (data corruption). Going smaller will ensure that we can find reasonable place where we feel Ok

[jatgarg]

Changed it to 90 sec. Should we reduce it more?

[vladsud]

Consider moving all connectivity-based logic into separate class as a pre-cursor to this change.
I believe this will allow you to

• write UTs independently
• I think (though not sure) it will be much easier to spell out states / transitions, and ideally not use promises - this will help in better understanding logic (asynchrony changes state and assumptions and it's in general very hard to see omissions, synchronous code forces to clearly articulate state when transition needs to happen). And I think it will lead to better code and better testability and hitting all conditions.

[jatgarg]

Added UTs too.

[vladsud]

Consider sprinkling telemetry events, such that when things go wrong, we have at least something to debug it.