Add TCP networking for target_os = "wasi" #1549

haraldh · 2022-02-15T12:29:09Z

With

merged, mio can have limited support for TCP networking for the wasm32-wasi target.

I also modified the tcp_server example.

$ cargo +nightly build --target wasm32-wasi --release --example tcp_server --features="os-poll net"
   Compiling cfg-if v1.0.0
   Compiling wasi v0.10.2+wasi-snapshot-preview1
   Compiling log v0.4.14
   Compiling libc v0.2.112
   Compiling ppv-lite86 v0.2.15
   Compiling wasi v0.11.0+wasi-snapshot-preview1
   Compiling getrandom v0.2.3
   Compiling rand_core v0.6.3
   Compiling env_logger v0.8.4
   Compiling rand_chacha v0.3.1
   Compiling mio v0.8.0 (/home/harald/git/mio)
   Compiling rand v0.8.4
    Finished release [optimized] target(s) in 2.92s

$ wasmtime run --tcplisten 127.0.0.1:9000 --env 'LISTEN_FDS=1' target/wasm32-wasi/release/examples/tcp_server.wasm
Using preopened socket FD 3
You can connect to the server using `nc`:
 $ nc <IP> <PORT>
You'll see our welcome message and anything you type will be printed here.

Thomasdezeeuw · 2022-02-15T17:21:17Z

Have you actually tested this? Because last I remember my code didn't really work.

haraldh · 2022-02-15T17:42:19Z

yes I did 😀

haraldh · 2022-02-17T09:28:25Z

Corrected the tcp_server example build and run instructions.

src/sys/wasi/net.rs

Thomasdezeeuw · 2022-02-18T14:54:27Z

@haraldh I just tried the TCP server example, but it doesn't work. Have you tried running the non-wasm version? The non-wasm returns a response on each packet of bytes send, but the wasm only responds when the connection is closed. When using nc you see a response every time yo hit enter (which sends the line you entered), but that doesn't work for the wasm version.

haraldh · 2022-02-18T15:55:05Z

@haraldh I just tried the TCP server example, but it doesn't work. Have you tried running the non-wasm version? The non-wasm returns a response on each packet of bytes send, but the wasm only responds when the connection is closed. When using nc you see a response every time yo hit enter (which sends the line you entered), but that doesn't work for the wasm version.

You are totally right... how about now with stream.set_nonblocking(true) ??

    pub(crate) fn accept(
        listener: &net::TcpListener,
    ) -> io::Result<(net::TcpStream, net::SocketAddr)> {
        let (stream, addr) = listener.accept()?;
        stream.set_nonblocking(true)?;
        Ok((stream, addr))
    }

@Thomasdezeeuw should now work as expected.

Thomasdezeeuw · 2022-02-18T16:01:31Z

@haraldh please don't force push during a review.

haraldh · 2022-02-18T16:04:41Z

Oops. sorry, didn't know one was happening..

Thomasdezeeuw

Their are some more problems with this stil.

Can't use read and write Interest in a single register call. This is a major issue.
Waker not implemented.
Selector::try_clone is broken.
Still needs UDP implementation.
Multiple threads not supported. Not a super big issue I think.

examples/tcp_server.rs

Thomasdezeeuw · 2022-02-18T19:22:08Z

I put some review notes and change in Thomasdezeeuw@e530779.

npmccallum · 2022-02-18T19:23:55Z

Their are some more problems with this stil.

Can't use read and write Interest in a single register call. This is a major issue.

Waker not implemented.

Selector::try_clone is broken.

Still needs UDP implementation.

Multiple threads not supported. Not a super big issue I think.

WebAssembly has no threads. Therefore, at least some of these issues don't apply. For example, what is the point of implementing Waker if there is only one thread?

WASI has no UDP support. Only streams are supported at this time.

Thomasdezeeuw · 2022-02-18T19:41:04Z

Their are some more problems with this stil.

Can't use read and write Interest in a single register call. This is a major issue.

Waker not implemented.

Selector::try_clone is broken.

Still needs UDP implementation.

Multiple threads not supported. Not a super big issue I think.

WebAssembly has no threads. Therefore, at least some of these issues don't apply. For example, what is the point of implementing Waker if there is only one thread?

Isn't that something in the pipeline? I don't remember the details of it, but I thought it was, but last I checked was a couple of years ago.

WASI has no UDP support. Only streams are supported at this time.

Ok.

So then we have:

Can't use read and write Interest in a single register call. This is still a major issue.
~~Waker not implemented.~~ We'll need to remove the public type.
Selector::try_clone is broken. We could remove the implementation for now.
~~Still needs UDP implementation.~~
Multiple threads not supported. Not a super big issue I think.

Could you cfg out the things that don't make sense for wasm, like the Waker type.

npmccallum · 2022-02-18T19:55:23Z

Their are some more problems with this stil.

Can't use read and write Interest in a single register call. This is a major issue.

Waker not implemented.

Selector::try_clone is broken.

Still needs UDP implementation.

Multiple threads not supported. Not a super big issue I think.

WebAssembly has no threads. Therefore, at least some of these issues don't apply. For example, what is the point of implementing Waker if there is only one thread?

Isn't that something in the pipeline? I don't remember the details of it, but I thought it was, but last I checked was a couple of years ago.

The spec isn't finalized yet. Portions of the proposal have been implemented in some runtimes. But WASI has no official way to spawn yet.

haraldh · 2022-02-21T16:29:22Z

@Thomasdezeeuw thanks for your review. I merged your PR and added additional cfg according to your comments.

Also added more cfg, so that cargo hack test --feature-powerset also works with:

$ CARGO_TARGET_WASM32_WASI_RUNNER="wasmtime run" cargo +nightly hack test --target wasm32-wasi --feature-powerset

src/net/tcp/listener.rs

examples/udp_server.rs

examples/tcp_server.rs

src/sys/wasi/mod.rs

haraldh · 2022-03-01T14:46:10Z

pushed more commits to be squashed in the end

npmccallum · 2022-03-07T20:07:23Z

@Thomasdezeeuw Any chance we can merge this? Are we waiting for something?

Thomasdezeeuw

One final nit, but otherwise LGTM.

If you don't get to the final change @haraldh I'll do it tonight and merge the pr.

Thomasdezeeuw · 2022-03-08T10:49:45Z

examples/tcp_server.rs

@@ -1,19 +1,17 @@
 // You can run this example from the root of the mio repo:
 // cargo run --example tcp_server --features="os-poll net"
-use mio::event::Event;


Can you revert all changes here, the changes make the example much harder to read. Please use the same structure as for the UDP example, with the extra main. I know it will generate a number of warnings, but that's OK.

yeah, will do

Based on tokio-rs#1395 And with * bytecodealliance/wasmtime#3711 * rust-lang/rust#93158 merged, mio can have limited support for networking for the `wasm32-wasi` target. Co-authored-by: Thomas de Zeeuw <thomasdezeeuw@gmail.com> Signed-off-by: Harald Hoyer <harald@profian.com>

Signed-off-by: Harald Hoyer <harald@profian.com>

haraldh · 2022-03-08T17:34:48Z

@Thomasdezeeuw squashed and left the old examples alone

haraldh · 2022-03-08T17:38:54Z

oh... you wanted it like UDP.. will change the examples.

Otherwise `cargo test` fails. Signed-off-by: Harald Hoyer <harald@profian.com>

Use the `LISTEN_FDS` mechanism to use pre-opened sockets. Especially for `wasm32-wasi` there is no other way to get access to sockets, than to use pre-opened sockets. Because `wasm32-wasi` does not yet return `TcpListener::local_addr()`, an unspecified IP address and port will be returned and displayed. ``` $ cargo +nightly build --target wasm32-wasi --release --example tcp_listenfd_server --features="os-poll net" Compiling cfg-if v1.0.0 Compiling wasi v0.10.2+wasi-snapshot-preview1 Compiling log v0.4.14 Compiling libc v0.2.112 Compiling ppv-lite86 v0.2.15 Compiling wasi v0.11.0+wasi-snapshot-preview1 Compiling getrandom v0.2.3 Compiling rand_core v0.6.3 Compiling env_logger v0.8.4 Compiling rand_chacha v0.3.1 Compiling mio v0.8.0 (/home/harald/git/mio) Compiling rand v0.8.4 Finished release [optimized] target(s) in 2.92s $ wasmtime run --tcplisten 127.0.0.1:9000 --env 'LISTEN_FDS=1' target/wasm32-wasi/release/examples/tcp_listenfd_server.wasm Using preopened socket FD 3 You can connect to the server using `nc`: $ nc <IP> <PORT> You'll see our welcome message and anything you type will be printed here. ``` Signed-off-by: Harald Hoyer <harald@profian.com>

haraldh · 2022-03-08T17:54:12Z

ok, ready to be merged, except if you want to squash all commits.

npmccallum · 2022-03-08T19:28:22Z

@Thomasdezeeuw Thanks! When can we get this in a release so that we can start working on the higher level components?

Thomasdezeeuw · 2022-03-08T19:33:44Z

Thanks @haraldh and @npmccallum for the work (and the patience).

@Thomasdezeeuw Thanks! When can we get this in a release so that we can start working on the higher level components?

Yes, I would like to include #1551 as well, so I think we can make a release this week.

Rustin170506 · 2022-03-19T02:45:04Z

I can't seem to run it in a non-wasm environment anymore. How should I run this example on macos?

Thomasdezeeuw · 2022-03-19T10:00:35Z

Did you follow the instruction in the example?

mio/examples/tcp_listenfd_server.rs

Lines 4 to 5 in e077a23

    
           // cargo +nightly build --target wasm32-wasi  --example tcp_listenfd_server --features="os-poll net" 
        
           // wasmtime run --tcplisten 127.0.0.1:9000 --env 'LISTEN_FDS=1' target/wasm32-wasi/debug/examples/tcp_server.wasm

That run for me on macOS. If it doesn't work for you can you provide some more details such as the error you're seeing.

haraldh · 2022-03-19T11:11:07Z

Oh, yeah the non-wasi version needs some update.

Rustin170506 · 2022-04-13T13:03:19Z

What is the purpose of our setting LISTEN_FDS here? We don't seem to be using this value.

Thomasdezeeuw · 2022-04-13T13:41:49Z

What is the purpose of our setting LISTEN_FDS here? We don't seem to be using this value.

The $LISTEN_FDS env var is a commonly used init systems, such as systemd, to indicate opened file descriptors (outside of standard in/out/err).

haraldh force-pushed the wasi_mio_upstream branch 2 times, most recently from d821baa to 10a6222 Compare February 15, 2022 12:42

haraldh force-pushed the wasi_mio_upstream branch from 10a6222 to 121d6ff Compare February 17, 2022 09:22

haraldh changed the title ~~Add networking for target_os = "wasi"~~ Add TCP networking for target_os = "wasi" Feb 17, 2022

npmccallum reviewed Feb 17, 2022

View reviewed changes

src/sys/wasi/net.rs Outdated Show resolved Hide resolved

haraldh force-pushed the wasi_mio_upstream branch from 121d6ff to 32158ae Compare February 18, 2022 15:53

Thomasdezeeuw reviewed Feb 18, 2022

View reviewed changes

examples/tcp_server.rs Outdated Show resolved Hide resolved

haraldh force-pushed the wasi_mio_upstream branch from 32158ae to 85fcddf Compare February 21, 2022 16:22

haraldh force-pushed the wasi_mio_upstream branch from 85fcddf to e883424 Compare February 21, 2022 16:33

haraldh requested a review from Thomasdezeeuw February 22, 2022 06:55

Thomasdezeeuw requested changes Feb 25, 2022

View reviewed changes

haraldh mentioned this pull request Mar 1, 2022

feat: remove the limitation of either R or W polls bytecodealliance/wasmtime#3866

Merged

Thomasdezeeuw reviewed Mar 1, 2022

View reviewed changes

src/sys/wasi/mod.rs Outdated Show resolved Hide resolved

src/sys/wasi/mod.rs Outdated Show resolved Hide resolved

Thomasdezeeuw approved these changes Mar 8, 2022

View reviewed changes

haraldh and others added 2 commits March 8, 2022 18:27

Disable all tests, which don't pass for target_os = "wasi"

be8120e

Signed-off-by: Harald Hoyer <harald@profian.com>

haraldh force-pushed the wasi_mio_upstream branch from 2fd6719 to db6652f Compare March 8, 2022 17:33

haraldh added 2 commits March 8, 2022 18:53

Stub non-working examples for target_os = "wasi"

2624922

Otherwise `cargo test` fails. Signed-off-by: Harald Hoyer <harald@profian.com>

haraldh force-pushed the wasi_mio_upstream branch from db6652f to 2b8b022 Compare March 8, 2022 17:53

Thomasdezeeuw approved these changes Mar 8, 2022

View reviewed changes

Thomasdezeeuw merged commit b2bbc10 into tokio-rs:master Mar 8, 2022

Thomasdezeeuw mentioned this pull request Mar 11, 2022

Release v0.8.1 #1552

Merged

MediosZ mentioned this pull request May 8, 2022

Create a Tokio-like async runtime in WasmEdge WasmEdge/WasmEdge#1429

Closed

haraldh deleted the wasi_mio_upstream branch November 22, 2022 09:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add TCP networking for target_os = "wasi" #1549

Add TCP networking for target_os = "wasi" #1549

haraldh commented Feb 15, 2022 •

edited

Loading

Thomasdezeeuw commented Feb 15, 2022

haraldh commented Feb 15, 2022

haraldh commented Feb 17, 2022

Thomasdezeeuw commented Feb 18, 2022

haraldh commented Feb 18, 2022

Thomasdezeeuw commented Feb 18, 2022

haraldh commented Feb 18, 2022

Thomasdezeeuw left a comment

Thomasdezeeuw commented Feb 18, 2022

npmccallum commented Feb 18, 2022 •

edited

Loading

Thomasdezeeuw commented Feb 18, 2022 •

edited

Loading

npmccallum commented Feb 18, 2022

haraldh commented Feb 21, 2022

haraldh commented Mar 1, 2022

npmccallum commented Mar 7, 2022

Thomasdezeeuw left a comment

Thomasdezeeuw Mar 8, 2022

haraldh Mar 8, 2022

haraldh commented Mar 8, 2022

haraldh commented Mar 8, 2022

haraldh commented Mar 8, 2022

npmccallum commented Mar 8, 2022

Thomasdezeeuw commented Mar 8, 2022

Rustin170506 commented Mar 19, 2022

Thomasdezeeuw commented Mar 19, 2022

haraldh commented Mar 19, 2022

Rustin170506 commented Apr 13, 2022

Thomasdezeeuw commented Apr 13, 2022

Add TCP networking for target_os = "wasi" #1549

Add TCP networking for target_os = "wasi" #1549

Conversation

haraldh commented Feb 15, 2022 • edited Loading

Thomasdezeeuw commented Feb 15, 2022

haraldh commented Feb 15, 2022

haraldh commented Feb 17, 2022

Thomasdezeeuw commented Feb 18, 2022

haraldh commented Feb 18, 2022

Thomasdezeeuw commented Feb 18, 2022

haraldh commented Feb 18, 2022

Thomasdezeeuw left a comment

Choose a reason for hiding this comment

Thomasdezeeuw commented Feb 18, 2022

npmccallum commented Feb 18, 2022 • edited Loading

Thomasdezeeuw commented Feb 18, 2022 • edited Loading

npmccallum commented Feb 18, 2022

haraldh commented Feb 21, 2022

haraldh commented Mar 1, 2022

npmccallum commented Mar 7, 2022

Thomasdezeeuw left a comment

Choose a reason for hiding this comment

Thomasdezeeuw Mar 8, 2022

Choose a reason for hiding this comment

haraldh Mar 8, 2022

Choose a reason for hiding this comment

haraldh commented Mar 8, 2022

haraldh commented Mar 8, 2022

haraldh commented Mar 8, 2022

npmccallum commented Mar 8, 2022

Thomasdezeeuw commented Mar 8, 2022

Rustin170506 commented Mar 19, 2022

Thomasdezeeuw commented Mar 19, 2022

haraldh commented Mar 19, 2022

Rustin170506 commented Apr 13, 2022

Thomasdezeeuw commented Apr 13, 2022

haraldh commented Feb 15, 2022 •

edited

Loading

npmccallum commented Feb 18, 2022 •

edited

Loading

Thomasdezeeuw commented Feb 18, 2022 •

edited

Loading