AccountsDb: make write_accounts_to_cache() never block sending to the bg hasher

[alessandrod]

Avoid commit_transactions() => store() => store_accounts_to() callers having to notify the bg hasher thread by doing an explicit sleep instead of sleeping on channel.recv() when the channel is empty. This avoids the case in which multiple replay/banking threads call commit_transactions() at the same time and end up... sleeping themselves acquiring the mutex to wake up the bg hasher.

Right side of the marker is with this PR

[alessandrod]

Here's a run I did a few weeks ago at higher throughput

[brooksprumo]

Can you help me understand, please?

On the store side, here's where we send the accounts to the background thread for hashing:

agave/accounts-db/src/accounts_db.rs

Lines 6531 to 6534 in c11fe18

	// hash this accounts in bg
	if let Some(ref sender) = &self.sender_bg_hasher {
	let _ = sender.send(cached_accounts);
	};

Are you saying that send() blocks these threads because we are calling recv() instead of try_recv() in the background hasher thread here?

agave/accounts-db/src/accounts_db.rs

Line 2347 in c11fe18

let result = receiver.recv();

[brooksprumo]

Or is the issue because send() will notify receivers, and using recv() is what registers a SyncReceiver?

notify receivers in send():
https://docs.rs/crossbeam-channel/latest/src/crossbeam_channel/flavors/list.rs.html#295-296

grab a lock to wake up a receiver:
https://docs.rs/crossbeam-channel/latest/src/crossbeam_channel/waker.rs.html#223

[brooksprumo]

(assuming my understanding in #3814 (comment) is correct)

[brooksprumo]

I'm not sure what the "right" sleep duration should be here, but 5 ms seems fine to start with.

[alessandrod]

Or is the issue because send() will notify receivers, and using recv() is what registers a SyncReceiver?

notify receivers in send(): https://docs.rs/crossbeam-channel/latest/src/crossbeam_channel/flavors/list.rs.html#295-296

grab a lock to wake up a receiver: https://docs.rs/crossbeam-channel/latest/src/crossbeam_channel/waker.rs.html#223

correct, the issue is racing to notify, sleeping on the lock

[mergify]

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.