feat: cacheless amt iteration

[LesnyRumcajs]

This exposes an alternative iteration method without caching. Without it, the memory usage for iterating over deal proposals is too high (my 64 GiB machine gets knocked out under a minute). For reference, this works fine in Go's AMT implementation.

With for_each_cacheless, I can iterate the proposals in ~90s (no-op iteration).

The logic is mostly taken from the for_each_while_mut.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.56%. Comparing base (9273717) to head (602ac18).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2189      +/-   ##
==========================================
+ Coverage   77.50%   77.56%   +0.06%     
==========================================
  Files         147      147              
  Lines       15743    15789      +46     
==========================================
+ Hits        12201    12247      +46     
  Misses       3542     3542              

Files with missing lines	Coverage Δ
ipld/amt/src/amt.rs	83.63% <100.00%> (+0.74%)	⬆️
ipld/amt/src/node.rs	87.56% <100.00%> (+1.22%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LesnyRumcajs]

CI blocked by #2187

[rvagg]

The only concern I have here is that this reportedly reads twice as many blocks as the cached version which seems off to me. I'd like to figure out what's going on there first.

[LesnyRumcajs]

Three iterations in the tests lead to this block read count.

1. Dirty cache - no block reads at all
   

   ref-fvm/ipld/amt/tests/amt_tests.rs

   Lines 418 to 424 in bf6fc44

   	// Iterate over amt with dirty cache
   	let mut x = 0;
   	a.for_each_cacheless(|_, _: &BytesDe| {
   	x += 1;
   	Ok(())
   	})
   	.unwrap();

2. 1st pass

ref-fvm/ipld/amt/tests/amt_tests.rs

Lines 434 to 445 in bf6fc44

	new_amt
	.for_each_cacheless(|i, _: &BytesDe| {
	if i != indexes[x] {
	panic!(
	"for each cacheless found wrong index: expected {} got {}",
	indexes[x], i
	);
	}
	x += 1;
	Ok(())
	})
	.unwrap();

3. 2nd pass

ref-fvm/ipld/amt/tests/amt_tests.rs

Line 448 in bf6fc44

new_amt.for_each_cacheless(|_, _: &BytesDe| Ok(())).unwrap();

Now, given that the only block read operation (and cache read) occurs here, it makes sense.

ref-fvm/ipld/amt/src/node.rs

Lines 451 to 457 in bf6fc44

	Link::Cid { cid, cache: _ } => {
	let node = bs
	.get_cbor::<CollapsedNode<V>>(cid)?
	.ok_or_else(|| Error::CidNotFound(cid.to_string()))?
	.expand(bit_width)?;
	node.for_each_cacheless(bs, height - 1, bit_width, offs, f)?;

All entries are "normally" cached on the 1st pass, but in the for_each_cacheless there is none, so the 2nd pass has to re-read all blocks (and not grab it from the cache) as it is here:

ref-fvm/ipld/amt/src/iter.rs

Lines 230 to 236 in bf6fc44

	match cache.get_or_try_init(|| {
	self.blockstore
	.get_cbor::<CollapsedNode<V>>(cid)?
	.ok_or_else(|| Error::CidNotFound(cid.to_string()))?
	.expand(self.bit_width)
	.map(Box::new)
	}) {

[rvagg]

needs docs, but lgtm
can be rebased on master because the other PR is merged now

[LesnyRumcajs]

warning: this URL is not a hyperlink
 --> ipld/amt/src/lib.rs:8:5
  |
8 | //! https://github.com/ipld/specs/blob/51fab05b4fe4930d3d851d50cc1e5f1a02092deb/data-structures/vector.md
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: bare URLs are not automatically turned into clickable links
  = note: `#[warn(rustdoc::bare_urls)]` on by default
help: use an automatic link instead
  |
8 | //! <https://github.com/ipld/specs/blob/51fab05b4fe4930d3d851d50cc1e5f1a02092deb/data-structures/vector.md>
  |     +                                                                                                     +

[LesnyRumcajs]

There are other doc links issues in the repo; I'll have a look later and see if there can be a friendly CI check to ensure this doesn't happen in the future.

[LesnyRumcajs]

#2191

[rvagg]

nice
I bet the HAMT could get exactlty the same treatment