feat: import_verifiable_stream and write_verifiable_stream in Store
#10
Conversation
|
I still requested a review just to confirm: Is this a feature we want to support in |
|
Documentation for this PR has been generated and is available at: https://n0-computer.github.io/iroh-blobs/pr/10/docs/iroh_blobs/ Last updated: 2025-01-03T14:55:42Z |
There was a problem hiding this comment.
I find it a bit weird that this is defined on MapEntry.
Wouldn't it be better to have this as a store fn that uses MapEntry? MapEntry really does everything it needs to do.
Also, it would be cool if we could make this more generic, otherwise we will have a dozen methods for special cases, like in this case streaming from offset..end of the file.
| fn import_verifiable_stream<'a>( | ||
| &'a self, | ||
| hash: Hash, | ||
| total_size: u64, |
There was a problem hiding this comment.
Where does the total_size come from? Is it a verified size?
|
write_verifiable_stream seems identical to https://docs.rs/iroh-blobs/latest/iroh_blobs/provider/fn.send_blob.html except that send_blob is more generic. |
|
Quick update on this: there are going to be a lot of changes to the store before 1.0. I am not happy with the current trait hierarchy at all. But this functionality will exist in some way in the new api. |
|
That's perfect. I'm happy to change iroh-willow to use any eqivalent new abstractions once they're in (doesn't have to be this PR). |
Basically I want to get rid of the entire trait hierarchy and replace it with a "protocol" that you use to talk to the store. There will be 2 protocol calls ImportBao and ExportBao, and these will then be wrapped in a pleasant way by an API that sits in front of the protocol. This is the bao impex part of the protocol: /// Export the given sizes in bao format, with the iroh block size.
///
/// The returned stream should be verified by the store.
#[derive(Debug)]
pub struct ExportBao {
pub hash: Hash,
pub ranges: ChunkRanges,
pub out: mpsc::Sender<EncodedItem>,
}
#[derive(Debug)]
pub struct ImportBao {
pub hash: Hash,
pub size: u64,
pub data: mpsc::Receiver<BaoContentItem>,
pub out: oneshot::Sender<anyhow::Result<()>>,
}And this is how it will be exposed in the "pleasant" API: pub async fn import_bao_quinn(
&self,
hash: Hash,
ranges: ChunkRanges,
stream: &mut quinn::RecvStream,
) -> anyhow::Result<()> { ... }
pub async fn import_bao_bytes(
&self,
hash: Hash,
ranges: ChunkRanges,
data: Bytes,
) -> anyhow::Result<()> { ... } |
|
Closing. Will keep track of this in #61 |
Description
Store::import_verifiable_streamMapEntry::write_verifiable_streamThese allow interacting with the
iroh_blobsstore using iroh's bao-style streams.I'm coming across this while extracting
iroh-willowinto its own thing. This means merging in all work that's not yet merged into iroh yet from thewillowPR/branch in iroh. This is one of these things.IIUC, this allows us to stream the willow entry payloads directly from the WGPS protocol instead of having to simultaneously run iroh-blobs & iroh-willow.
Breaking Changes
Adding fn to traits should be non-breaking as long as they have default impls, right?
Notes & Open Questions
Unfortunately no tests with this yet, other than my pinky-promise that this code has worked so far for iroh-willow.
That said, I haven't tried bigger payloads yet.
I'll probably make use of this branch for the iroh-willow extraction and come back eventually to add proper tests.
Change checklist