feat: align wasi-http with component linker

[eduardomourar]

In the context of wasi-http, this PR will use the shared functionality from wasmtime-wasi will prepare the ground for the migration from the preview2-prototyping repo to wasmtime. A table is being defined to simulate the future resource type. Additionally, it will simplify using Component Linker with the wasi-http module.

Usage example with WASI Preview2

use wasmtime::{
    component::{Component, Linker},
    Config, Engine, Store,
};
use wasmtime_wasi::preview2::{
    command::Command, Table, WasiCtx, WasiCtxBuilder, WasiView,
};
use wasmtime_wasi_http::{WasiHttpCtx, WasiHttpView};

pub struct Ctx {
   table: Table,
    wasi: WasiCtx,
    http: WasiHttpCtx,
}

impl WasiView for Ctx {
    fn table(&self) -> &Table {
        &self.table
    }
    fn table_mut(&mut self) -> &mut Table {
        &mut self.table
    }
    fn ctx(&self) -> &WasiCtx {
        &self.wasi
    }
    fn ctx_mut(&mut self) -> &mut WasiCtx {
        &mut self.wasi
    }
}

impl WasiHttpView for Ctx {
    fn http_ctx(&self) -> &WasiHttpCtx {
        &self.http
    }
    fn http_ctx_mut(&mut self) -> &mut WasiHttpCtx {
        &mut self.http
    }
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let wasi = WasiCtxBuilder::new().inherit_stdio().build();
    let http = WasiHttpCtx::new();
    let mut config = Config::new();
    config.wasm_backtrace_details(wasmtime::WasmBacktraceDetails::Enable);
    config.wasm_component_model(true);
    config.async_support(true);

    let engine = Engine::new(&config)?;
    let component = Component::from_file(&engine, "component.wasm")?;
    let mut store = Store::new(&engine, Ctx { table, wasi, http });
    let mut linker = Linker::new(&engine);

    add_to_linker(&mut linker)?;
    wasmtime_wasi_http::add_to_component_linker(&mut linker)?;

    let (wasi, _instance) =
        wasi::Command::instantiate_async(&mut store, &component, &linker).await?;
    let result = wasi
        .call_main(&mut store, &[])
        .await
        .map_err(|e| anyhow::anyhow!("wasm failed with {e:?}"))?
        .map_err(|e| anyhow::anyhow!("command returned with failing exit status {e:?}"))?;

    Ok(())
}

Depends on: #6836.

[eduardomourar]

@pchickey and @brendandburns , could you review this PR, please? Ideally, we should merge this prior to #6091.

[brendandburns]

Is there a reason for the rename?

[eduardomourar]

just to follow the pattern in the other modules, and also to show that there is a breaking change to the API

[brendandburns]

I think it is probably worth splitting this into two PRs, one containing all of the common implementations and one with the refactors to the HTTP implementation.

Using the common code for things like streams seems reasonable.

I don't really see the point of the HTTP refactors, they seem to add a bunch of complexity without really reducing the amount of code that is required.

[eduardomourar]

I don't really see the point of the HTTP refactors, they seem to add a bunch of complexity without really reducing the amount of code that is required.

This is important to integrate with preview2, but also to have less code change whenever the resources will be auto-generated by wit-bindgen. We will be a step further in that direction instead of diverging more and more from the expected pattern.

[pchickey]

This looks to me like a reasonable path to align with some of the changes coming down the pipe from preview2-prototyping. It doesn't make those changes totally drop-in, but using the streams and table abstraction does take us part of the way there.

I agree that, without that context, it looks like added complexity without reducing the amount of code required.

So really the question is whether adding this now is appropriate, or if we try to land Brendan's other patches first. Unfortunately I am still deep in the preview 2 filesystem work over in p2-p and I haven't context switched back to HTTP to review Brendan's PRs yet. #6228 is pretty straightforward but #6091 has been sitting for longer and I haven't dug into it yet.

[alexcrichton]

I'm going to list @pchickey as the reviewer for this since I think he's got this in his wheelhouse

[eduardomourar]

@pchickey , I can understand if we want to first merge the other PRs. After they are, I will integrate the changes back here then.

[brendandburns]

Just for my own understanding, I don't think that anything in http_impl.rs changes with the move to preview 2 does it? That struct already implements the Host trait that is defined by the wasi-http with bindings. Why does switching to Table make it more compatible? I think I'm missing something.

[eduardomourar]

I would expect the http_impl.rs to be modified, yes. If you look into this example here and other parts of the host implementation, you will see that the table abstraction is being assumed to be available and it is the default place to store the id for each handle and associated resource instance. By doing the same for the HTTP module in this PR, we would only hold specific data in its context (right now there is none) and rely on the common context instead. This simplification is not that evident here because I copied the shared code to avoid creating a dependency to the preview2-prototyping repo.

[pchickey]

Another wrinkle is that we'll be spreading this implementation across multiple crates. wasi-common will define the Table, a trait WasiInputStream , and then hook up all the input-stream methods in the io.streams wit to lookup in the table and downcast it to a Box<dyn WasiInputStream>, and then invoke the method.

wasmtime-wasi-http's role will be to define struct HttpBodyInputStream( {whatever the right tokio/hyper primitives here} ), impl wasi_common::WasiInputStream for HttpBodyInputStream, and then creation of a new stream will be to cast it to a Box and stick it into a table, which is shared with the wasi_common::WasiCtx.

We may end up needing to pull some additional tricks to get wasi-common and wasmtime-wasi-http to each access the table via &mut Table instead of using an Arc<Mutex<Table>> to handle sharing, but that is just an optimization we can do once the basics work.

[brendandburns]

It seems like much of the refactor is unrelated to streams, I get that consistent stream handling makes sense, but doesn't seem like using a Table to store the request/response objects is inherently better than having a struct with multiple hashtables inside of it. Indeed it seems to complicate the implementation instead.

Moreover, given that there are multiple other wasi-modules in wasmtime that are even further from being component-linker compatible, I'd rather we focused energy on getting the HTTP implementation correct in terms of user-facing surface area and wait for preview2 to arrive in wasmtime before we worry about these kind of changes.

[eduardomourar]

The problem is that currently we cannot use a Wasm component (+ preview2) in different runtimes (in my case wasmtime, Node.js and browser). I guess other people/companies are facing the same issue.

The way I see, we have two options:

• we go forward with this PR (after the pending PRs are merged, obviously) by bringing preview2 partially here;
• or we move all this logic to the preview2-prototyping repo;

I actually preferred the second option, which was the one decided originally. Although, you can take this PR as a compromise where we will temporarily maintain those in two places in order to bring the learnings upstream and they will eventually converge.