Optimize Watches

[sfc-gh-mpilman]

FDB Watches currently are not as cheap as they could be. We should consider making the following optimizations:

• If the client creates multiple watches on the same key, they should be collapsed, so we only pay for them once
• Same thing on the server: instead of having one actor per watch, we should have at most one actor per key and this actor can inform multiple clients of a change
• If the client fails, or if it cancels a watch, we currently don't cancel this watch on the server. We should consider doing that though, as especially clients that fail often can be a burden for a storage

[sfc-gh-ljoswiak]

A client cancelling a watch should cause a "soft" cancel. A soft cancel means instead of sending an RPC to the storage server telling it to cancel the watch, a flag will be set on the watch. This flag will instruct the watch to not renew itself when it times out on the storage server, and to not deliver any messages if triggered. On timeout, the watch should be cleaned up.

[sfc-gh-mpilman]

The motivation for what Lukas' said is the following:
If a client creates a watch, then cancels a watch and later recreates the watch, we will send two RPCs to the storage. With the proposed change we would only send one. Keeping the future from the server-side watch around should be relatively low overhead

[sfc-gh-abeamon]

With the proposed change we would only send one

I believe that setting a watch also involves reading the key being watched. This may still be necessary in the proposed change to avoid the case where there is a race between you re-enabling a watch and the watch triggering due to a change preceding your transaction.

[sfc-gh-mpilman]

Yes, a client would still need to read it.

Basically we will first need a change to roll forward the version at which a watch is on the client side (if a value didn't change between versions x and y, we can move the version forward). This will still need a few design discussions though...

[sfc-gh-nwijetunga]

Benchmarks measuring these improvements can be found in this doc