Generalize wasm-builder for arbitrary targets

[mrcnski]

ISSUE

Overview

For paritytech/polkadot-sdk#650 we want to build one of the crates in polkadot as a separate musl binary, as part of the polkadot build process, and then include the binary in polkadot itself.

The functionality would essentially be identical to wasm-builder but with a different target. Looking through the code, most of it doesn't seem specific to wasm, apart from "wasm" being used in many of the names. Nevertheless, we should be able to adapt it for arbitrary targets.

Alternatives considered

1. I would rather not copy/paste all the wasm-builder code (1600+ lines) as it would be awful for maintainability.
2. I spent several hours trying to do a purely cargo-based solution, and have come to the conclusion that it's not possible due to its severely limited functionality. It doesn't even support post-build scripts. Also, with my (not working) cargo solution we would have to hardcode the triple x86_64-unknown-linux-musl, but we may want to support a different arch in the future.
3. Would be better not to introduce alternative build tools so we can keep building everything with just cargo build.

[bkchr]

Yes in general we could do this, but I'm not sure that the wasm-builder is the right place for this. Moving shareable parts into a different crate could be a possible solution for this. However, as the general direction of the the PVF worker is not finally decided, I would not start working on this yet.

[mrcnski]

Moving shareable parts into a different crate could be a possible solution for this.

Agreed, had the same thought after submitting the issue.

[35359595]

We have problems building some primitives (sp-core dependant) for sgx target because of strict check for 32 bit words for non-std builds (where's our target is x64). Will this problem be part of this refactoring or should I raise separate issue for it? @bkchr

[mrcnski]

As of now my use case is no longer relevant:

For paritytech/polkadot-sdk#650 we want to build one of the crates in polkadot as a separate musl binary, as part of the polkadot build process, and then include the binary in polkadot itself.

This will now be part of the build guide for validators and not automated. So maybe this issue can be closed.

[bkchr]

We have problems building some primitives (sp-core dependant) for sgx target because of strict check for 32 bit words for non-std builds (where's our target is x64). Will this problem be part of this refactoring or should I raise separate issue for it? @bkchr

You want this: #13982 (comment)

[35359595]

We have problems building some primitives (sp-core dependant) for sgx target because of strict check for 32 bit words for non-std builds (where's our target is x64). Will this problem be part of this refactoring or should I raise separate issue for it? @bkchr

You want this: #13982 (comment)

Wrong URL? :)

[bkchr]

Ohh 😅

#5547 now :D