feat: Implement Block Height Stream #776
Conversation
darunrs
force-pushed
the
block-streamer-blockheight-stream
branch
from
eed39fc to
71d6c59
Compare
block-streamer/src/bitmap.rs
Outdated
| if self.data.get_bit(self.bit_index) { | ||
| self.bit_index += 1; | ||
| return Some( | ||
| self.data.start_block_height + u64::try_from(self.bit_index - 1).ok()?, |
There was a problem hiding this comment.
In what scenario would we not be able to parse this? Is returning None really the right thing to do here?
There was a problem hiding this comment.
Why not just make bit_index u64?
There was a problem hiding this comment.
So bit_index needs to be usize to index into the vector. I could still make it u64 but then I need to convert it to usize eventually when doing vector[index]. Converting from usize to u64 shouldn't ever fail anyway as the largest value a bit_index should ever be is 86400 anyway. Even if we change bitmap to index by year, it would be roughly 32 million, which easily fits inside a 32bit uint.
I also agree that returning None is strange. Converting a u64 to usize is "trickier" since usize could be a u32 depending on the architecture. But converting a usize to u64 should never fail and truncate data since most architectures are 64bit anyway. I'll just use the "as" keyword in this instance.
| @@ -1,65 +1,101 @@ | |||
| use anyhow::anyhow; | |||
There was a problem hiding this comment.
Great work on this, the separated structs makes this a lot easier to understand :)
I've implemented a struct which is capable of returning an async stream which yields block heights matching a particular contract filter starting from a specified block height. This should serve as a replacement for the existing method of getting block heights by querying Delta Lake through the Delta Lake Client.
Some differences between Delta Lake and this new stream is that it yields a SINGLE block height rather than all block heights. In addition, we now query the bitmap indexer's tables for block heights for one day, instead of for all days. This should improve time to get a block as well as reduce the memory footprint.