Gateway: Fetch recursive DAG as CAR #304
Conversation
iroh-resolver/src/resolver.rs
Outdated
| let mut cids = VecDeque::new(); | ||
| let this = self.clone(); | ||
| async_stream::try_stream! { | ||
| let root_block = this.resolve_root(&root).await.map(|(cid, loaded)| OutRaw::from_loaded(cid, loaded)); |
There was a problem hiding this comment.
I think this is the same code as resolve_recursive effectively, other than mapping things differently, maybe this could be abstracted by passing a closer to a shared function?
There was a problem hiding this comment.
Sounds good. I implemented an abstraction to be shared between the two resolve_recursive functions. resolve_recursive_with_path doesn't yet use it because there's additional logic around keeping the path. Could add it to the generic one, if always maintaining the path isn't considered too expensive (don't think it would matter).
There was a problem hiding this comment.
@dignifiedquire I've looked into having the remaining resolve_recursive_with_path also use the resolve_recursive_mapped function from the latest commit in this PR. I made it work but with the cost of a couple of additional clones of the path even if it's not being used. See this commit (not on this PR): Frando@91c0440
I wasn't sure if it's worth it and couldn't find a way to avoid the clones (because of lifetime issues due to async move in the closure)
also fixes clippy and code style issues
|
I added a test as well. @Arqu because there's not any integration tests in the gateway yet maybe review if the testing approach is correct. |
|
Lovely, thank you. The testing story is a bit grim currently besides some smaller bits. There's a pending issue I have to tackle to get tests going but will have to wait a little bit longer. |
Hi!
I was looking on how to use iroh from other codebases, and a bit of a precursor to a writable gateway, this PR adds functionality to fetch a DAG recursively into a CAR file. This was already stubbed in the gateway but not implemented.
The implementation works similar to the existing recursive loaders. However to save the cost of reserializing, I added a new
OutRawstruct in the resolver for this (because theOutstruct in the resolver currently only keeps the deserialized representation for UnixFS nodes). Currently, the nodes are still fully deserialized anyways for link scraping but that would be fixed with n0-computer/beetle#162.What's still missing is, I think, a configurable upper limit on DAG sizes that can be requested from the Gateway to avoid DoS attacks. Also misses tests, I can add some if the PR seems on the right track to you.