Cannot run the WASM demos

[fgm]

What version of CUE are you using (cue version)?

$ cue version
cue version v0.0.0-20240307083748-cdcb6fa19bf3

go version go1.22.0
      -buildmode exe
       -compiler gc
  DefaultGODEBUG httplaxcontentlength=1,httpmuxgo121=1,tls10server=1,tlsrsakex=1,tlsunsafeekm=1
     CGO_ENABLED 1
          GOARCH arm64
            GOOS darwin
             vcs git
    vcs.revision cdcb6fa19bf332283500711ce5381df1cc1935e7
        vcs.time 2024-03-07T08:37:48Z
    vcs.modified true

Does this issue reproduce with the latest stable release?

Yes, but slightly differently.

What did you do?

• followed the example starting at https://docs.google.com/presentation/d/1PaAbydkiG8fp2mapIOYedw_UY1sHz7QG-T7-4tI9e9A/edit#slide=id.g255965d3018_0_71
• lib.rs is the one in the example
• Cargo.toml is this:

[package]
name = "hello"
version = "0.1.0"
edition = "2021"

[dependencies]

[lib]
crate-type = ["cdylib"]

• I built in two different ways, generating the hello.wasm both times, and tried the commands after that one with both version, with identical results
  • cargo build --target wasm32-wasi
  • cargo build --target wasm32-unknown-unknown
• copy the hello.wasm file to the project directory containing the wasmer.cue file:

@extern("wasm")
package p

add: _ @extern("hello.wasm", abi=c, sig="func(int64, int64): int64")
isPrime: _ @extern("hello.wasm", abi=c, name=is_prime, sig="func(uint64): bool")

x0: add(1, 2)
x1: add(-1, 2)
x2: add(100, 1)
b1: isPrime(127)
b2: isPrime(128)

• ran cue eval wasmer.cue, with no flags and various flags combinations, both with the cue (current stable) and cue08 (master HEAD) binary
  • consistent result :

can't load from external module: load "hello.wasm": file not found:
    ./wasmer.cue:4:8
can't load from external module: load "hello.wasm": file not found:
    ./wasmer.cue:5:12

• ran cue eval wasmer.cue hello.wasm, with no flags and various flags combinations, both with the cue (current stable) and cue08 (master HEAD) binary
  • with master HEAD: same error as above
  • with current stable (0.7.1): unknown file extension .wasm
• just to be sure, ran the cue test suite and the tests for the wasm package pass locally
• I also ran the same commands with sudo fs_usage -f pathname cue08 in another window to trace file system calls

What did you expect to see?

• normal results
• in the fs_usage output, at least a stat or open to the file hello.wasm possibly at an incorrect path to explain the errors

What did you see instead?

• errors above
• in the fs_usage window, after the prologue:

stat64            .                                                                                0.000005
stat64            /Users/fgm/Dropbox/Formation/Go/Cue/external                                     0.000005
stat64            .                                                                                0.000005
stat64            /Users/fgm/Dropbox/Formation/Go/Cue/external                                     0.000005
stat64            .                                                                                0.000003
stat64            /Users/fgm/Dropbox/Formation/Go/Cue/external                                     0.000002
stat64            /Users/fgm/Dropbox/Formation/Go/Cue/external/cue.mod                             0.000006
stat64            /Users/fgm/Dropbox/Formation/Go/Cue/external/cue.mod                             0.000002
open              /Users/fgm/Dropbox/Formation/Go/Cue/external/cue.mod/module.cue                  0.000027
close                                                                                              0.000003
stat64            /Users/fgm/Dropbox/Formation/Go/Cue/external/wasmer.cue                          0.000004
open              /Users/fgm/Dropbox/Formation/Go/Cue/external/wasmer.cue                          0.000015
close                                                                                              0.000001
open              /Users/fgm/Dropbox/Formation/Go/Cue/external/wasmer.cue                          0.000007
close                                                                                              0.000001
stat64            .                                                                                0.000002
stat64            /Users/fgm/Dropbox/Formation/Go/Cue/external                                     0.000002```

Showing there was no attempt to stat or open the hello.wasm file.

What more did you do ?

I ran cue master HEAD under debugger, and found out:

• the attempt to load the @extern("hello.wasm") imports happen in wasm.findFile from the list of resolved files in the was minstance’s b.UnknownFiles, which is empty.
• that list is populated from b.OrphanFiles in cmd.getDecoders but that list is empty too.
• b.OrphanFiles is populated in load.fp.add but that function only ever receives the wasmer.cue file and not the hellow.wasmfile, regardless of whether it was passed on the CLI.
• one thing I noticed is that my hello.wasm is much larger than those in the test suite, at 170kB vs less that 1kB for the small ones and 15kB for the largest one. Possibly due to the presence of debug info, I'm not very familiar with Rust builds.

Anything else ?

• I am following this Rust path to evaluate WASM integration, but if I have to use WASM, I would prefer to use gc or tinygo because the functions I need to use are written in Go, and it would avoid having to rewrite them in Rust
• the whole reason to do this is because for one project I need a bunch of network related functions operating on IP addresses, masks, CIDR info, etc, which is larger than the existing functions pkg/net, especially regarding the mixed IPv4/IPv6 situations.
• I am exploring this WASM solution because the current mechanism is to
  • fork cue
  • add our functions in a net2 Go package
  • add the package to pkg/packages.txt
  • go generate ./pkg
  • add that package to pkg/register.go until registration becomes automatic autogenerated as in the TODO in gen.go#24
  • go build -o cue08 cuelang.org/go/cmd/cue
• We would very much prefer a dynamic solution that does not require maintaining a Cue fork, but that would possibly use something like plugins, either the (shudder) Go stdlib version, https://github.com/hashicorp/go-plugin or even something using a RPC service like gRPC.

[myitcv]

Thanks, @fgm - I've copied in @4ad

[4ad]

Hmm, can you try without using CUE modules?

[4ad]

To answer your ancillary questions more specifically:

I am following this Rust path to evaluate WASM integration, but if I have to use WASM, I would prefer to use gc or tinygo because the functions I need to use are written in Go, and it would avoid having to rewrite them in Rust

Gc is impossible to use because it can't export functions yet (see golang/go/#65199).

Tinygo is not guaranteed to work because it doesn't have a stable, or a standard ABI. Realistically only C and Rust can be expected to work at this point.

Additionally, there is no support for strings yet, which I imagine you might need.

• I am exploring this WASM solution because the current mechanism is to
  • fork cue
  • add our functions in a net2 Go package
  • add the package to pkg/packages.txt
  • go generate ./pkg
  • add that package to pkg/register.go until registration becomes automatic autogenerated as in the TODO in gen.go#24
  • go build -o cue08 cuelang.org/go/cmd/cue
• We would very much prefer a dynamic solution that does not require maintaining a Cue fork

Do you need to use the CUE command line tool, or would a Go API to inject additional functions suffice? We could potentially make it easy to add functions via the API, but of course, if you need to use the command line tool rather than the API this would not help you.

[fgm]

@4ad Thanks for answering so fast. I get the same behavior if I remove the module (cue.mod/), that was how I initially did it, actually : I only added the cue.mod/ to check if that fixed things.

Using the CUE CLI would be best, but using a Go API to inject additional functions might probably be sufficient: the devs in that team mostly use the CLI to debug their configurations, but the actual logic using the validations is in custom Go code, so that should not be a major issue.

I noticed you mention C and Rust: AIUI gccgo can work with the C ABI : is that correct and could that be a solution ?

[4ad]

I might be wrong, but I don't believe gccgo supports Wasm at all.

Additionally, the "C ABI" for Wasm is not defined anywhere really, so every compiler does something slightly different. Sometimes even the same compiler can do something different (see wasm32-wasi vs. wasm32-unknown-unknown above). That's the problem with Wasm, really, and the reason why we want to move to the Component Model as soon as it is feasible.

[fgm]

OK, so it would be either the existing fork-based solution (short term), or a way to add custom functions in Go for custom apps (second best, if you think that is something which is easily doable), or WASM from Rust once we figure out why I cannot run it yet, then.

Is there an existing issue I could look-at/help-with for the second solution ?

[fgm]

OK, so I found the problem: i was running, as shown: cue eval wasmer.cue or cue08 eval wasmer.cue, and got the errrors above.

Running simply cue eval, cue08 eval, cue eval ., or cue08 eval . there is no problem, the code works as expected. That was similar to running a single Go file vs running a package.

I found the info by dissecting what basic.txtar actually did.

[4ad]

Ah, yes, that makes perfect sense.

[fgm]

FWIW, -buildmode=c-shared i was just accepted in golang/go#65199 so it's now only a matter of time before Go can export functions. So maybe that will mean the ability to use Go-sourced WASM for functions in some weeks/months.

[4ad]

Kind of. The Go compiler does not use the C ABI, so we will need to add a Go-specific ABI, it wouldn't Just Work. But yes.

[fgm]

Hypothetically, any idea of the amount of work it would take should I try to build this to Cue ?

[4ad]

The problem is that the only way to get the memory layout of Go structs is via the -asmhdr flag of the Go compiler. Since we need this information it would most likely mean we need to compile the Go code on the user's behalf, since we can't expect the user to provide us with this information.

We also need a way for this information to find its way into CUE. Perhaps we could add it to CUE struct fields as an attribute. Of course we don't want users to have to write this by hand, so we would need some sort of converter from the -asmhdr output to CUE.

On a technical side of things, making use of this information (once we have it in CUE) is probably quite easy. The more difficult part is making an ergonomic UI/UX around it for users, and plumbing all the pieces together.

In other words the difficulty lies in designing the feature rather than implementing it.

[fgm]

@4ad you just marked this as completed : is it just routine maintenance, or any specific change ? Related to #2035 AFAICS.

[4ad]

Apologies, I wrote a message but it somehow got erased when I closed the issue.

This wasn't really a bug as Wasm was intended to be used with full packages and not with individual files.

But we have removed Wasm support from the CUE command itself (API support still present), so this issue doesn't even serve as a feature request for the forseeable future. We're waiting for Component Model support and when (if) that will arrive it will necessarily have a different UX in CUE.