http POST support #499

fabricedesre · 2022-11-15T23:22:52Z

@Arqu I could get some help with something in the http handler: the new put_handler itself compiles fine but anytime I use the unixfs builder in that function (uncommenting any of the let file = create_file_from_xxx line) I get a trait bound error from the route setup that I don't understand:

error[E0277]: the trait bound `fn(Extension<Arc<core::State<T>>>, http::Request<hyper::Body>) -> impl futures::Future<Output = Result<response::GatewayResponse, GatewayError>> {put_handler::<T>}: Handler<_, _>` is not satisfied
   --> iroh-gateway/src/handlers.rs:71:33
    |
71  |         .route("/:scheme/", put(put_handler::<T>))
    |                             --- ^^^^^^^^^^^^^^^^ the trait `Handler<_, _>` is not implemented for `fn(Extension<Arc<core::State<T>>>, http::Request<hyper::Body>) -> impl futures::Future<Output = Result<response::GatewayResponse, GatewayError>> {put_handler::<T>}`
    |                             |
    |                             required by a bound introduced by this call
    |
    = help: the trait `Handler<T, ReqBody>` is implemented for `Layered<S, T>`
note: required by a bound in `axum::routing::put`
   --> /home/fabrice/.cargo/registry/src/github.com-1ecc6299db9ec823/axum-0.5.17/src/routing/method_routing.rs:402:1
    |
402 | top_level_handler_fn!(put, PUT);
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `axum::routing::put`
    = note: this error originates in the macro `top_level_handler_fn` (in Nightly builds, run with -Z macro-backtrace for more info)

Thanks!

…nt (n0-computer#317) use store exported path in iroh-one closes #255, closes #309

* feat(gateway): fetch a DAG recursively as CAR. * test(gateway): add test for recursive car fetch also fixes clippy and code style issues * refactor(resolver): use mapper in recursive resolve * style(resolver): fix comment * style(resolver): clippy fix * style(gateway): clippy fix and test fix

* fix(unixfs): ensure links get persisted in store This had to be merged with some other changes related to the size info. Introduce a Block struct that has data, hash and links for DRY. * test(unixfs): validate blocks in all roundtrip tests Validate blocks in all roundtrip tests to make sure that the blocks that are returned from file.encode or dir.encode are internally consistent. * fix(store): only create new id when needed The old code was creating new ids even if we already knew the cid, which caused some havoc.

This allows adding large files without getting an error. It is a stopgap until we have a persistent kad store.

## move RPC client construction into `iroh` (read: `iroh-api`) I've moved creation of the RpcClient away from the CLI into `iroh` (to be `iroh-api`). This means that the API gains a bunch of dependencies and the CLI loses them. Doing this made me realize that the api should own the rpc client, so changed from a reference (with a lifetime) to ownership. ## `iroh get` start of implementation Start of implementation of `iroh get` with a few mocked CLI tests ## `iroh` -> `iroh-api` crate The plan is to have an `iroh` crate that contains the CLI and an `iroh-api` crate that contains the Rust API. Here we've done half of that renaming: `iroh` is renamed to `iroh-api`. I haven't done the second half, renaming `iroh-ctl` to `iroh`, as that would make this commit too different to read.

* Rename the iroh-ctl crate to the iroh crate * test: test --version flag does what it should now

Acknowledges that a `raw` node is a valid `UnixfsNode` and therefore can be resolved as a file in the gateway

- implements rabin based chunking (following kubo & dagger) -expose chunker selection on the api & cli - improve the builder api for files

crates.io release prep

fixes for issues found during the 0.1.2 release

Arqu

I've reviewed this mostly to keep the conversation going.
@b5 what's our roadmap/product stance on the writable gateway story?

Arqu · 2022-11-30T09:56:31Z

iroh-gateway/src/handlers.rs

+ // If this gateway is not writable, return a 400 error.
+ if !state.config.writeable_gateway() {
+ return Err(GatewayError::new(
+ StatusCode::BAD_REQUEST,


Instead of 400 Bad Request we should do 405 Method Not Allowed

Arqu · 2022-11-30T09:56:40Z

iroh-gateway/src/handlers.rs

+#[tracing::instrument(skip(state))]
+pub async fn post_handler<T: ContentLoader + std::marker::Unpin>(
+ Extension(state): Extension<Arc<State<T>>>,
+ // Path(params): Path<HashMap<String, String>>,


Let's drop unused code.

Arqu · 2022-11-30T09:59:51Z

iroh-gateway/src/handlers.rs

+ ));
+ }
+
+ // TODO: check path & headers


We should check for valid scheme, all the headers. We should also check for bad_bits in the denylist, but given the logistics we would probably first need to add it and then based on the CID respond and remove it if its in the list.

Arqu · 2022-11-30T10:02:13Z

iroh-resolver/src/content_loader.rs

+ &self,
+ _content: T,
+ ) -> Result<cid::Cid, anyhow::Error> {
+ unimplemented!()


This is still unimplemented. Should we allow this?
On a separate note, feels wrong to be placed in the content loader. Unsure if the content loader is evolving to be more (and whether it should) or if we should move this out and have it be handled by the gateway client and not the resolver (my preference).

Arqu · 2022-11-30T10:09:10Z

iroh-resolver/src/unixfs_builder.rs

@@ -168,6 +170,8 @@ pub struct File {
 chunker: Chunker,
 }

+unsafe impl Send for File {}


Not thrilled about unsafe code. @dignifiedquire what's our general policy on that front?

No, this is not safe to do. Each piece of the struct needs to be made Send

dignifiedquire · 2022-12-07T12:43:25Z

iroh-resolver/src/chunker.rs

@@ -35,6 +35,8 @@ pub enum ChunkerStream<'a> {
 Rabin(LocalBoxStream<'a, io::Result<Bytes>>),
 }

+unsafe impl<'a> Send for ChunkerStream<'a> {}


No, this is not safe to do. The streams need to change to BoxStream to make this work

dignifiedquire · 2022-12-07T12:44:01Z

iroh-resolver/src/unixfs_builder.rs

 Path(PathBuf),
 }

+unsafe impl Send for Content {}


this should not be needed, otherwise something is going wrong

dignifiedquire

None of the unsafe usages are actually safe to use, these need reworking

fabricedesre · 2022-12-07T17:57:52Z

@dignifiedquire I'm not surprised the unsafe parts need to be re-done, it was mostly "compiler error" based implementation.

I'm not convinced by the overall design anyway. Like @Arqu comments in https://github.com/n0-computer/iroh/pull/499/files#r1035765976 we should settle on that question first.

dignifiedquire and others added 30 commits October 10, 2022 20:50

fixup some session management

01f130f

Merge remote-tracking branch 'origin/main' into bitswap2

ac2639f

use session ids from resolver

4bbdd57

fix(iroh-one): use default store path when no --store-path flag prese…

643fd43

…nt (n0-computer#317) use store exported path in iroh-one closes #255, closes #309

fix: Update ipld in iroh-car to limit prost duplicates (n0-computer#307)

ee43bd7

Merge remote-tracking branch 'origin/main' into bitswap2

fe0b3ba

fix race condition in decision in engine

c3def87

fix(p2p): Increase the size of the kad mem store (n0-computer#320)

00f5660

This allows adding large files without getting an error. It is a stopgap until we have a persistent kad store.

fix(resolver): stop sessions when requests are done

9c46cb0

feat: rename the iroh-ctl crate to the iroh crate (n0-computer#321)

8b02977

* Rename the iroh-ctl crate to the iroh crate * test: test --version flag does what it should now

hash received blocks

72c0c09

avoid cloning MessageQueue

b256608

message_queue: extract loopstate

6c3700b

message_queue: remove locking

20125a6

Merge remote-tracking branch 'origin/main' into bitswap2

4d837a9

fix: allow Raw as UnixfsType (n0-computer#327)

275d88b

Acknowledges that a `raw` node is a valid `UnixfsNode` and therefore can be resolved as a file in the gateway

optimize wantlist

c6cf412

session_want_sender: avoid ref counting

bf3c869

session: avoid callbacks

785284f

cleanup client ownership

7fa6a8d

add missing await

a7ce238

dont print full message

d9b836f

record outgoing blocks

8ccc9f7

record better message stats

9250f66

record successfull dials

8b8d8cd

record found providers

10915c8

improve logging in error cases

5449b86

dignifiedquire and others added 20 commits November 28, 2022 16:04

feat(unixfs): add rabin based chunking

b0a5783

- implements rabin based chunking (following kubo & dagger) -expose chunker selection on the api & cli - improve the builder api for files

chore: update version deps

d0d74f0

docs(readme): add badges

a686090

docs(readme): add badges

48724a1

chore: do not publish examples

c314191

chore: ensure release commits are consolidated

0dab051

chore: add RELEASE.md

0ad846a

docs(release): add required permissions

b780237

Merge pull request n0-computer#560 from n0-computer/release-prep

b57cf13

crates.io release prep

chore(release): add changelog for v0.1.2

9f9ab27

chore(cargo): fixup Cargo.toml files

9d650b0

chore: Release

58eeb29

fix(iroh): do not use self version dep

6f6fee0

fix(iroh): exclude tests and fixtures from publishing

6ebee5f

fix: version handling in tests & git_version

59c5dc7

chore(iroh-resolver): ignore tests and fixtures on publish

078d23c

Merge pull request n0-computer#561 from n0-computer/fixes-for-relase

cdfb710

fixes for issues found during the 0.1.2 release

chore(release): v0.1.3 changelog

13a7f0c

chore: Release

5702c3e

Initial http POST support

8469b5e

fabricedesre force-pushed the http-put branch from 4ee55b4 to 8469b5e Compare November 29, 2022 22:49

fabricedesre changed the title ~~WIP: http PUT support~~ http POST support Nov 29, 2022

Arqu reviewed Nov 30, 2022

View reviewed changes

dignifiedquire reviewed Dec 7, 2022

View reviewed changes

dignifiedquire requested changes Dec 7, 2022

View reviewed changes

RangerMauve mentioned this pull request Jan 30, 2023

IPIP: Data Onboarding via HTTP POST (and future ipfs:// POST|PUT) ipfs/specs#375

Open

dignifiedquire closed this Feb 20, 2023

dignifiedquire force-pushed the main branch from 6527ee4 to 3a3fc46 Compare February 20, 2023 15:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http POST support #499

http POST support #499

fabricedesre commented Nov 15, 2022

Arqu left a comment

Arqu Nov 30, 2022

Arqu Nov 30, 2022

Arqu Nov 30, 2022

Arqu Nov 30, 2022 •

edited

Loading

Arqu Nov 30, 2022

dignifiedquire Dec 7, 2022

dignifiedquire Dec 7, 2022

dignifiedquire Dec 7, 2022

dignifiedquire left a comment

fabricedesre commented Dec 7, 2022

http POST support #499

http POST support #499

Conversation

fabricedesre commented Nov 15, 2022

Arqu left a comment

Choose a reason for hiding this comment

Arqu Nov 30, 2022

Choose a reason for hiding this comment

Arqu Nov 30, 2022

Choose a reason for hiding this comment

Arqu Nov 30, 2022

Choose a reason for hiding this comment

Arqu Nov 30, 2022 • edited Loading

Choose a reason for hiding this comment

Arqu Nov 30, 2022

Choose a reason for hiding this comment

dignifiedquire Dec 7, 2022

Choose a reason for hiding this comment

dignifiedquire Dec 7, 2022

Choose a reason for hiding this comment

dignifiedquire Dec 7, 2022

Choose a reason for hiding this comment

dignifiedquire left a comment

Choose a reason for hiding this comment

fabricedesre commented Dec 7, 2022

Arqu Nov 30, 2022 •

edited

Loading