Post compilation error when adding additional substrate crate

[ivanceras]

Using sp-core works without problem, but then adding the additional substrate crate: sp-runtime, sp-version, sp-std, sp-keyring will make a compilation issue.

our fork in branch substrate-integration.

$ yarn rebuild
yarn run v1.22.10
warning package.json: No license field
$ yarn clean && yarn codegen && yarn build
warning package.json: No license field
$ rm -rf module/src/w3 && rm -rf test/w3
warning package.json: No license field
$ npx w3 codegen
✔ Manifest loaded from ./web3api.yaml
✔ Codegen Web3API schema bindings
🔥 Types were generated successfully 🔥
warning package.json: No license field
$ npx w3 build
Error: Invalid Wasm module found. `mutation` at <home>/integrations-chainsafe/protocol/substrate/core-wrapper/build/mutation.wasm is invalid. Error: ,Error: Unsupported wasm import namespace requested: "__wbindgen_placeholder__"; Supported wasm import namespaces: "env", "w3"
    at Compiler.<anonymous> (<home>/integrations-chainsafe/protocol/substrate/core-wrapper/node_modules/@web3api/cli/build/lib/Compiler.js:680:31)
    at step (<home>/integrations-chainsafe/protocol/substrate/core-wrapper/node_modules/@web3api/cli/build/lib/Compiler.js:54:23)
    at Object.throw (<home>/integrations-chainsafe/protocol/substrate/core-wrapper/node_modules/@web3api/cli/build/lib/Compiler.js:35:53)
    at rejected (<home>/integrations-chainsafe/protocol/substrate/core-wrapper/node_modules/@web3api/cli/build/lib/Compiler.js:27:65)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Primary investigation suggest that this is cause by usage of wasm-bindgem which is coming from getrandom crate.

Take note that the getrandom crate was included beforehand in the crates' Cargo.toml together with sp-core and it works without problems. It's when the other 4 crates: sp-runtime, sp-version, sp-std, sp-keyring where it makes the compilation issue occur.

[dOrgJelli]

I've fixed the issue in the following PR: https://github.com/ChainSafe/integrations/pull/1/files

Fix

The steps to fix this can be found in the build steps listed within the Dockerfile build image:

1. Install wasm-bindgen:

RUN cargo install -f wasm-bindgen-cli

2. Enable the "WASM_INTERFACE_TYPES" feature, which will remove the __wbindgen_throw import. See: https://github.com/rustwasm/wasm-bindgen/blob/7f4663b70bd492278bf0e7bba4eeddb3d840c868/crates/cli-support/src/lib.rs#L397-L403 :

ENV WASM_INTERFACE_TYPES=1

3. Run wasm-bindgen over the module, replacing all placeholder __wbindgen_... imports:

RUN wasm-bindgen ./module/target/wasm32-unknown-unknown/release/module.wasm --out-dir ./build --out-name bg_module.wasm

4. Run wasm-snip and wasm-opt as normal

Debugging Information

After looking at the module.wasm file in WAT format (human-readable text) I found the following imports:

(import "__wbindgen_placeholder__" "__wbindgen_describe" (func $__wbindgen_placeholder__.__wbindgen_describe (type $t4)))
(import "__wbindgen_externref_xform__" "__wbindgen_externref_table_grow" (func $__wbindgen_externref_xform__.__wbindgen_externref_table_grow (type $t6)))
(import "__wbindgen_externref_xform__" "__wbindgen_externref_table_set_null" (func $__wbindgen_externref_xform__.__wbindgen_externref_table_set_null (type $t4)))
(import "__wbindgen_placeholder__" "__wbindgen_throw" (func $__wbindgen_placeholder__.__wbindgen_throw (type $t2)))

I tried to find ways to remove these imports, and realized that a wasm-bindgen CLI exists: https://rustwasm.github.io/docs/wasm-bindgen/reference/cli.html

I then deduced that these strange imports are probably "placeholder" imports that are later processed by the wasm-bindgen CLI. I decided to then try and run the wasm-bindgen CLI over the wasm file. Success! The imports shown above were removed, and were replaced with a single import __wbindgen_throw:

(import "./bg_module.wasm_bg.js" "__wbindgen_throw" (func $./bg_module.wasm_bg.js.__wbindgen_throw (type $t1)))

Why only a single "throw" import? This is probably because none of the types that are being used with wasm-bindgen ever cross the wasm-boundary, so their serialization methods do not needs to be added.

Now we just need to find a way to remove this single import, as it isn't necessary. I opened up the wasmbindgen source code: https://github.com/rustwasm/wasm-bindgen

A simple ctrl+f for __wbindgen_throw revealed this section of code:
https://github.com/rustwasm/wasm-bindgen/blob/7f4663b70bd492278bf0e7bba4eeddb3d840c868/crates/cli-support/src/throw2unreachable.rs#L7-L8

Perfect! Now we just need to figure out how to get this routine to run. Looks like the WASM_INTERFACE_TYPES environment variable is being used for that, see this entry-point function here:
https://github.com/rustwasm/wasm-bindgen/blob/7f4663b70bd492278bf0e7bba4eeddb3d840c868/crates/cli-support/src/lib.rs#L98

After enabling this flag, and running wasm-bindgen, we now have a "clean" wasm module with all wasm-bindgen functionality stripped away!