Tracking issue for building IPA against wasm-unknown-unknown target

[akoshelev]

According to https://github.com/cloudflare/workers-rs this is required to run IPA on CF Workers

[akoshelev]

executing

rustup target add wasm32-unknown-unknown
cargo build --target wasm32-unknown-unknown

[akoshelev]

brace for impact

cargo build --target wasm32-unknown-unknown
   Compiling typenum v1.15.0
   Compiling mio v0.8.5
   Compiling wasm-bindgen-macro-support v0.2.83
   Compiling slab v0.4.7
   Compiling hashbrown v0.12.3
   Compiling futures-channel v0.3.25
   Compiling futures-task v0.3.25
   Compiling futures-util v0.3.25
   Compiling futures-io v0.3.25
   Compiling bitflags v1.3.2
error[E0432]: unresolved import `crate::sys::IoSourceState`
  --> /Users/koshelev/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.5/src/io_source.rs:12:5
   |
12 | use crate::sys::IoSourceState;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^ no `IoSourceState` in `sys`

error[E0432]: unresolved import `crate::sys::tcp`
  --> /Users/koshelev/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.5/src/net/tcp/listener.rs:15:17
   |
15 | use crate::sys::tcp::{bind, listen, new_for_addr};
   |                 ^^^ could not find `tcp` in `sys`

error[E0432]: unresolved import `crate::sys::tcp`
  --> /Users/koshelev/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.5/src/net/tcp/stream.rs:13:17
   |
13 | use crate::sys::tcp::{connect, new_for_addr};
   |                 ^^^ could not find `tcp` in `sys`

[akoshelev]

Based on conversations here seanmonstar/reqwest#1300
tokio-rs/tokio#1597

and the way Reqwest deals with Tokio: https://github.com/seanmonstar/reqwest/blob/master/Cargo.toml#L159

my conclusion is we have to take the whole Tokio crate out in order to build for WASM. Tokio supports wasm-wasi target, but it does not support wasm-unknown-unknown

[akoshelev]

IPA uses tokio channels and runtime

grep -r "use tokio" . --include \*.rs
./benches/iai/arithmetic_circuit.rs:use tokio::runtime::Builder;
./benches/ct/arithmetic_circuit.rs:use tokio::runtime::Builder;
./src/test_fixture/transport/mod.rs:use tokio::sync::mpsc::{channel, Sender};
./src/test_fixture/transport/mod.rs:use tokio_stream::wrappers::ReceiverStream;
./src/test_fixture/transport/routing.rs:use tokio_stream::wrappers::ReceiverStream;
./src/net/server/handlers/query.rs:use tokio::sync::mpsc;
./src/lib.rs:    pub use tokio::task::JoinHandle;
./src/protocol/attribution/credit_capping.rs:    use tokio::try_join;
./src/helpers/old_network.rs:use tokio::sync::mpsc;
./src/helpers/old_network.rs:use tokio_util::sync::{PollSendError, PollSender};
./src/helpers/buffers/receive.rs:use tokio::sync::oneshot;
./src/helpers/error.rs:use tokio::sync::mpsc::error::SendError;
./src/helpers/error.rs:use tokio_util::sync::PollSendError;
./src/helpers/http/network.rs:use tokio::sync::mpsc;
./src/helpers/http/network.rs:use tokio_stream::wrappers::ReceiverStream;
./src/test_fixture/transport/routing.rs:use ::tokio::sync::{mpsc, oneshot};
./src/net/server/mod.rs:use ::tokio::sync::mpsc;
./src/helpers/messaging.rs:use ::tokio::sync::{mpsc, oneshot};

[akoshelev]

hmm, CF have added support for WASI recently: https://blog.cloudflare.com/announcing-wasi-on-workers/

[akoshelev]

wow, tokio is actually fine with wasm-wasi, ring compilation fails

PKG_CONFIG_SYSROOT_DIR=/ RUSTFLAGS="--cfg tokio_unstable" cargo build --target wasm32-wasi
   Compiling tracing v0.1.37
   Compiling futures-util v0.3.25
   Compiling serde v1.0.147
   Compiling openssl-sys v0.9.78
   Compiling http-body v0.4.5
   Compiling thiserror v1.0.37
   Compiling ring v0.16.20
   Compiling lock_api v0.4.9
   Compiling parking_lot_core v0.9.4
   Compiling lazy_static v1.4.0
   Compiling portable-atomic v0.3.15
error[E0432]: unresolved import `super::sysrand_chunk`
   --> /Users/koshelev/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/src/rand.rs:306:16
    |
306 |     use super::sysrand_chunk::chunk;
    |                ^^^^^^^^^^^^^ could not find `sysrand_chunk` in `super`

   Compiling unicode-xid v0.2.4
   Compiling once_cell v1.16.0
   Compiling regex-syntax v0.6.28
   Compiling openssl v0.10.43
   Compiling foreign-types-shared v0.1.1
For more information about this error, try `rustc --explain E0432`.
error: could not compile `ring` due to previous error
warning: build failed, waiting for other jobs to finish...

[martinthomson]

Setting wasm32_unknown_unknown_js as a ring feature might address this.

[akoshelev]

I've spent sometime researching on this topic, here is what I found

We basically have two ways to deploy IPA to CF Workers

Use workers-rs crate

This crate seems to be a wrapper around wasm-bindgen that provides bindings to CF Workers API.

Pros:

• Rust-native interface
• Routing can live inside the app and be written in 100% Rust code.

Cons:

• Tokio dependency must be eliminated for WASM build.
• People on Discord report some hard time dealing with workers-rs crate: https://discord.com/channels/595317990191398933/882757739019911219/1060610466839285884
• wasm-unknown-unknown support for various Rust crates is significantly worse than wasm32-wasi. We may incur some heavy tax on evolving IPA.

Work required to eliminate Tokio from WASM build:

• Replace tokio channels with something that is wasm-unknown-unknown compatible. We extensively use channels in infra, so I'd replace them entirely rather than making a compile-time switch.
  • candidates: https://docs.rs/futures-channel
• replace tokio::test attribute or ignore all tests for WASM builds.
• replace tokio::select. No good alternatives known to me
• replace tokio::time::sleep. We already do this for Shuttle, so no issues here, very easy to do.
• replace tokio::spawn with wasm_bindgen_futures::spawn_local. Same here. Maybe it is time to make spawn interface centralized for the entire IPA app (similar to rand/shuttle entry points)
• Move HTTP app to its own crate. Create a new crate to support CF workers. This is probably optional but I am not sure. I fear we end up having tons of conditional flags inside the HTTP module which may make it unreadable. Alternatively we can upgrade to axum 0.6 that supports WASM builds.

Use WASI

According to https://blog.cloudflare.com/announcing-wasi-on-workers/, there is an experimental support for WASI on CF workers. It requires IPA to compile for wasm32-wasi target.

Pros

• Tokio builds for this target. Less work/hacks inside IPA code to maintain two targets.
• Given the above, we get IPA running on CF Workers faster.

Cons

• I don't know whether we can keep tokio::spawn or need to replace it.
• We would need a Javascript wrapper to route requests, no routing support in Rust.
• It seems that the only way to communicate with a WASI IPA is through stdin/stdout

Note that it is probably possible to support native routing in Rust, but would require us to replicate the work done in workers crate.

Conclusion

If we decide that it is worth investing in CF workers support, I'd go with option 1 and use workers crate as it carries less risks in discovering something entirely unexpected and requiring to write bindings and/or hacks.

[cjpatton]

There's sort of two dimensions to consider here. The first is the software ecosystem for deploying rust on Workers, which for the moment is fairly immature. (As you saw.) FWIW, I have been using workers-rs for a while, and it seems to work well enough when you're developing your application specifically for that platform. For instance, Daphne has much of the core protocol logic in a separate rust crate daphne, which can be reused across platforms; and it also has a crate daphne_worker for the Workers-specific stuff.

The second question is whether Workers is a suitable architecture for this protocol. Beyond trying to get the code to build, have you spent any time looking into how the backend components would work? What is needed for long term storage, and how would you implement it? (There are lots of options here.)

[xtuc]

@akoshelev did you manage to get past the ring compilation issue with the wasm32-wasi target? I noticed the following issues on the ring repo:

• When can wasm32-wasi support？ briansmith/ring#1572
• wasm32-wasi doesn't build briansmith/ring#1043

Would it be possible to switch to another crypto library that has wasm32-wasi support?

A subset of Tokio can be used with wasm32-wasi. Since I can't compile the project i'm not able to tell if it's going to work on Cloudflare Workers. The WASI support on Workers is incomplete. For instance, WASI's poll_oneoff is not available which Tokio relies on in some cases.

[akoshelev]

@cjpatton apologies, somehow I missed the notification about this response.

I have been using workers-rs for a while, and it seems to work well enough when you're developing your application specifically for that platform. For instance, Daphne has much of the core protocol logic in a separate rust crate daphne, which can be reused across platforms; and it also has a crate daphne_worker for the Workers-specific stuff.

I agree with it, we did not realize the need to support CF workers until late 2022. Tokio crate brings more than just a scheduler so our code uses it extensively. It does still seem to be possible to use workers-rs crate, but the limitations I outlined above.

The second question is whether Workers is a suitable architecture for this protocol.

So far it does look promising for IPA use case. MPC helper state is ephemeral, does not require durable storage (mod helper configuration) and MPC computation is network heavy. There is an open question how to store the query results, but this operation is relatively infrequent for the duration of worker lifetime.

Summarizing it, Durable objects seem to cover our use cases for storage well, but ofc it remains to be seen whether DO is enough.

[akoshelev]

did you manage to get past the ring compilation issue with the wasm32-wasi target?

@xtuc I was some time ago, but think @martinthomson suggestion to use wasm32_unknown_unknown_js feature solved it for us.