3. fix(state): prevent watch channel deadlocks in the state

[teor2345]

Motivation

This PR makes watch channel deadlocks in the state impossible, which avoids deadlocks. (And time-consuming investigations into deadlock causes.)

It also avoids some livelocks, and some performance issues. (Clones are cheap, lock contention is expensive.)

Specifications

If the watch receiver holds a read lock, then the sender tries to get the write lock, then another receiver tries to get another read lock, the watch channel can deadlock.

See the "potential deadlock example" in:
https://docs.rs/tokio/latest/tokio/sync/watch/struct.Receiver.html#method.borrow

Designs

The WatchReceiver wrapper clones the watched data before allowing access to it, then explicitly drops the read lock as soon as possible. This prevents:

• read-write-read deadlocks, and
• read-write livelocks under heavy use,

because the read lock is dropped immediately after the clone.

For more details, see the conversation at #3847 (comment)

Solution

Deadlock prevention:

• Add a WatchReceiver wrapper that always clones the borrowed watch data
• Explicitly drop the read lock as quickly as possible

Memory optimisations:

• Drop the shared Arcs as quickly as possible

Ergonomics:

• Make read::block more flexible, by accepting any AsRef
• Make the block method docs consistent

Testing

When testing syncing lightwalletd from a zebrad that is also syncing from peers, I saw pauses of up to 3 minutes. These pauses finished when lightwalletd gave up and exited. The pauses seem to be fixed by commit 3240332.

Review

This is part of an high-priority series of state refactor PRs, because they will cause merge conflicts with other PRs.

Reviewer Checklist

• Code implements Specs and Designs
• Tests for Expected Behaviour
• Tests for Errors

Follow Up Work

• Testing syncing lightwalletd from zebrad syncing from peers (maybe Add lightwalletd full sync integration test #3511?)

[conradoplg]

Looks good

[teor2345]

@Mergifyio refresh

[mergify]

refresh

✅ Pull request refreshed

[codecov]

Codecov Report

Merging #3870 (50fcb7d) into main (641f488) will increase coverage by 0.09%.
The diff coverage is 92.10%.

❗ Current head 50fcb7d differs from pull request most recent head 3240332. Consider uploading reports for the commit 3240332 to get more accurate results

@@            Coverage Diff             @@
##             main    #3870      +/-   ##
==========================================
+ Coverage   78.68%   78.78%   +0.09%     
==========================================
  Files         296      297       +1     
  Lines       33833    33879      +46     
==========================================
+ Hits        26622    26690      +68     
+ Misses       7211     7189      -22     

[dconnolly]

Looks ok to me but I kinda would like @jvff to look at it.

[dconnolly]

No comments, since this is High 🔥 let's merge and hopefully @jvff can give a look if they are available