Standardised verifiable builds #1148

SkymanOne · 2023-06-06T16:05:33Z

Refers to #1065

This PR aims to bring the workflow to produce verifiable and reproducible builds using cargo contract --verifiable and docker container with the pre-defined based on Debian distro.

In detail, the command spawns the container with the binding volume to the contact folder (See image documentation for more details), then executes cargo contract build --release --output-json <some additional flags> > target/build_result.json to build the contract in the release mode and save the serialized BuildResult output to a file. We need that in order to allow the host machine to access the results and print them in the STDOUT in the requested format.

After the execution, the container and build_result.json files are deleted. The host then accesses metadata files, deserializes them back into the rust object, add the image name, and writes them back.

It was noted that building an image on different architectures (AMD64 vs ARM64) results in the production of different WASM binaries. Therefore, it was decided to stick with the single AMD64 image for the builds. While it may have an impact on the build times on ARM machines, this is the only way we can guarantee consistency in the reproducible builds.

The image has been published to: https://hub.docker.com/r/paritytech/contracts-verifiable

Since the docker image is supposed to track the cargo-contract use paritytech/contracts-verifiable:test tag which can be manually specified with the --image flag.

Once the PR is merged, master tag tracking master branch will be published.

The follow up PR will be focused on the CI to automate the image release process.

Tasks

Successfully spawn and remove containers upon execution
Persist image digest id in the metadata
Save output to the host folder
Respect cargo flags passed to the host
Allowing to specify docker image to use

Some extra goals I think worth considering

~~Allowing to specify docker image to use~~
Specify the docker command to be used for the build
Recognise path dependant multi-contract projects and mount directories automatically ( by @deuszx )

Points 2 and 3 can be considered as a follow-up issues

crates/build/src/lib.rs

SkymanOne · 2023-06-07T11:03:00Z

We need to approve the dockerfile first to be published on registry before releasing the cargo-contract. Once the image is approved and published, docker.rs will just pull it from registry instead of trying to find the local copy.

build-image/Dockerfile

build-image/README.md

ascjones

Some stray unwraps need removing, there might be some more.

crates/cargo-contract/src/cmd/extrinsics/call.rs

crates/build/src/docker.rs

crates/build/src/lib.rs

SkymanOne · 2023-06-08T16:26:33Z

Updated image. I used @deuszx dockerfile for inspiration. I managed to install cargo-contract from crates.io and still have a size of ~890Mb.

As I said earlier, we need to be consistent with the system architecture. I have compiled flipper contract using the ARM version of the image and the X86_64 one. The two produced binaries were different.

We need to approve the dockerfile of the image first, so I can contact CI team to publish it. Then, I will proceed with finishing off this PR.

@ascjones @deuszx

build-image/Dockerfile

deuszx · 2023-06-09T09:57:03Z

Can you add a CI step somewhere here where you build the same contract on multiple OSes and verify that their outcomes match?

crates/build/src/docker.rs

ascjones

LGTM

SkymanOne · 2023-07-13T16:03:02Z

The branch has been merged to master. Please use paritytech/contracts-verifiable:master image for now until the crate is published.

sourabhniyogi · 2023-07-13T17:52:50Z

Tested out paritytech/contracts-verifiable:master but must be doing it wrong:

# docker run -d --name ink-container --mount type=bind,source=".",target="/contract"  paritytech/contracts-verifiable:master
137b6f0a3aaa1de3d2ceb55314347b3d9596b39ceb3cef8ea889fa1bd807b4b6

# docker logs 137b6f0a3aaa1de3d2ceb55314347b3d9596b39ceb3cef8ea889fa1bd807b4b6
Utilities to develop Wasm smart contracts

Usage: cargo contract <COMMAND>

Commands:
  new          Setup and create a new smart contract project
  build        Compiles the contract, generates metadata, bundles both together in a `<name>.contract` file
  check        Check that the code builds as Wasm; does not output any `<name>.contract` artifact to the `target/` directory
  upload       Upload contract code
  instantiate  Instantiate a contract
  call         Call a contract
  encode       Encodes a contracts input calls and their arguments
  decode       Decodes a contracts input or output data (supplied in hex-encoding)
  remove       Remove contract code
  info         Display information about a contract
  help         Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

Can you advise?

SkymanOne · 2023-07-13T17:59:44Z

Tested out paritytech/contracts-verifiable:master but must be doing it wrong:

# docker run -d --name ink-container --mount type=bind,source=".",target="/contract"  paritytech/contracts-verifiable:master
137b6f0a3aaa1de3d2ceb55314347b3d9596b39ceb3cef8ea889fa1bd807b4b6

# docker logs 137b6f0a3aaa1de3d2ceb55314347b3d9596b39ceb3cef8ea889fa1bd807b4b6
Utilities to develop Wasm smart contracts

Usage: cargo contract <COMMAND>

Commands:
  new          Setup and create a new smart contract project
  build        Compiles the contract, generates metadata, bundles both together in a `<name>.contract` file
  check        Check that the code builds as Wasm; does not output any `<name>.contract` artifact to the `target/` directory
  upload       Upload contract code
  instantiate  Instantiate a contract
  call         Call a contract
  encode       Encodes a contracts input calls and their arguments
  decode       Decodes a contracts input or output data (supplied in hex-encoding)
  remove       Remove contract code
  info         Display information about a contract
  help         Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

Can you advise?

The entry point now is cargo contract, you need to specify the command to run
https://github.com/paritytech/cargo-contract/blob/db242993af92a9860fd753207cf9583e5a95a23e/build-image/Dockerfile#L111

Just bear in mind, if you run the build manually, the resulted metadata file won't contain image field as it is added as part of post-build step in cargo-contract

sourabhniyogi · 2023-07-13T22:15:24Z

I tried it out on 4 cases (where all 4 cases were generated from ChainIDE WASM Contract templates and compiled in ChainIDE), and flipper compiled but the other 3 did not. Errors are included below:

# docker run -d --name ink-container --mount type=bind,source="./src",target="/contract" paritytech/contracts-verifiable:master build
# docker logs [...]
 [1/*] Building cargo project
error: failed to parse manifest at `/tmp/cargo-contract_RqgRr5/Cargo.toml`

Caused by:
  the target `my_psp22` is a binary and can't have any crate-types set (currentlERROR: command ["/usr/local/rustup/toolchains/1.69-x86_64-unknown-linux-gnu/bin/cargo", "build", "--color=always", "--target=wasm32-unknown-unknown", "-Zbuild-std=core,alloc", "--no-default-features", "--release", "--target-dir=/contract/target/ink", "--features", "ink/ink-debug"] exited with code 101

shiden34

# docker run -d --name ink-container --mount type=bind,source="./src",target="/contract" paritytech/contracts-verifiable:master build
# docker logs [...]
Some errors have detailed explanations: E0405, E0408, E0412, E0425, E0433, E0463, E0531.
For more information about an error, try `rustc --explain E0405`.
error: could not compile `proc-macro2` due to 451 prevERROR: command ["/usr/local/rustup/toolchains/1.69-x86_64-unknown-linux-gnu/bin/cargo", "build", "--color=always", "--target=wasm32-unknown-unknown", "-Zbuild-std=core,alloc", "--no-default-features", "--release", "--target-dir=/contract/target/ink", "--features", "ink/ink-debug"] exited with code 101

factory_contract

# docker run -d --name ink-container --mount type=bind,source="./src",target="/contract" paritytech/contracts-verifiable:master build
# docker logs [...]
ERROR: Error invoking `cargo metadata` for Cargo.toml

Caused by:
    `cargo metadata` exited with an error:     Updating crates.io index
        Updating git repository `https://github.com/727-Ventures/openbrush-contracts`
    error: failed to get `pair_contract` as a dependency of package `factory_contract v0.1.0 (/contract)`
    
    Caused by:
      failed to load source for dependency `pair_contract`
    
    Caused by:
      Unable to update /pair
    
    Caused by:
      failed to read `/pair/Cargo.toml`
    
    Caused by:
      No such file or directory (os error 2)

We were happy that case 1 generated the same codehash as chainide but any insights to cases 2/3/4 would be appreciated!

SkymanOne · 2023-07-14T00:55:38Z

docker run -d --name ink-container --mount type=bind,source="./src",target="/contract" paritytech/contracts-verifiable:master build

I compiled the given contracts locally (in host environment), and got the same errors. The issue is unrelated to this PR. Please open respective issues in 727-Ventures/openbrush-contracts repo.

wufengtao1 · 2023-07-14T04:25:25Z

docker run -d --name ink-container --mount type=bind,source="./src",target="/contract" paritytech/contracts-verifiable:master build

I compiled the given contracts locally (in host environment), and got the same errors. The issue is unrelated to this PR. Please open respective issues in 727-Ventures/openbrush-contracts repo.

Hi. I am from Chainide. Based on my experience, the last three contracts require +nightly and rustc 1.67.1 to compile successfully (you can compile them successfully at https://staging-9589904a8a.chainide.com/s/dashboard/projects). The reason for this might be related to this: #1058. So, I believe either Openbrush should update their template or the Docker should support "cargo +nightly-2023-02-07 contract build". Either way, I think it requires communication among Openbrush, Paritytech, Chainide, and Polkaholic.

SkymanOne · 2023-07-14T14:37:12Z

@wufengtao1 @sourabhniyogi can you please open a separate issue in cargo-contract and ping the relevant stakeholders? The new version of the cargo-contract has not yet been released. So, we still have some time to tweak the image to accommodate all the stakeholders that require this feature.

wufengtao1 · 2023-07-18T03:02:44Z

@SkymanOne Sure. Let me handle this.

* docker image + tokio * docker client draft * refactoring execute func * successful docker build * persist flags in the main execution context * docker readme + respect host flags * add build steps * better image, custom image arg, container reuse * fix readme example * fix docs * update README * docs fixes * clarification in the docs * use locked for cargo install * concat install and cleanup commands * use arg instead of env in dockerfile * add version labels * update comments * decompose the function * correct argument passing in dockerfile * refactoring * more fixes * using remote registry + bell & whistles (progress bars) * changelog entry * fixes * refactoring * explicitly specify versions of apt packages * refactoring * add module docs * refactoring * doc fix and optionally use git in docker image * Error handling for builds * use single var in docker and refactor docker module * notes on apple silicon * remove unused arg in dockerfile * use target path correctly * handle building with relative paths + permissions * more flags in docker image + fixes * remove unused code * add git info to docker metadata * use env to detect running OS * detect host OS * non-root docker user * mount contract to home dir in docker * give permission to target to everyone * fix typo * use current uid and gid for docker * conditional compilation * dont cache ci template and use musl in docker * OS specific code blocks * print error logs * remove the message * calculate digest in separate func * filter our status msgs in error * update comments * resolve other merge conflicts * display container name * properly trancate error output * apply Andrews suggestions * proper build step logs * fix generation of ABI * hash the cmd, not entrypoint for digest * fix args parsing and add docs * clippy fix

SkymanOne added 5 commits June 5, 2023 15:10

docker image + tokio

b3eafd3

docker client draft

a11bb5b

refactoring execute func

ade7003

successful docker build

26328c6

persist flags in the main execution context

773a2ff

ascjones reviewed Jun 6, 2023

View reviewed changes

crates/build/src/lib.rs Outdated Show resolved Hide resolved

docker readme + respect host flags

763cc77

SkymanOne marked this pull request as ready for review June 7, 2023 11:01

SkymanOne requested review from cmichi and HCastano as code owners June 7, 2023 11:01

add build steps

300b725

deuszx reviewed Jun 7, 2023

View reviewed changes

build-image/Dockerfile Outdated Show resolved Hide resolved

deuszx reviewed Jun 7, 2023

View reviewed changes

build-image/README.md Show resolved Hide resolved

ascjones reviewed Jun 8, 2023

View reviewed changes

crates/cargo-contract/src/cmd/extrinsics/call.rs Outdated Show resolved Hide resolved

crates/build/src/docker.rs Outdated Show resolved Hide resolved

deuszx reviewed Jun 8, 2023

View reviewed changes

crates/build/src/docker.rs Outdated Show resolved Hide resolved

deuszx reviewed Jun 8, 2023

View reviewed changes

crates/build/src/lib.rs Show resolved Hide resolved

SkymanOne marked this pull request as draft June 8, 2023 11:58

better image, custom image arg, container reuse

7dca32b

SkymanOne requested review from ascjones and deuszx June 8, 2023 17:18

SkymanOne added 3 commits June 8, 2023 18:44

fix readme example

55b72f1

Merge branch 'master' into gn/verifiable-build

f63a53e

fix docs

9955f25

SkymanOne marked this pull request as ready for review June 8, 2023 18:23

update README

5fe03fb

xermicus reviewed Jun 8, 2023

View reviewed changes

build-image/Dockerfile Show resolved Hide resolved

build-image/Dockerfile Outdated Show resolved Hide resolved

build-image/Dockerfile Outdated Show resolved Hide resolved

ascjones reviewed Jun 9, 2023

View reviewed changes

build-image/Dockerfile Outdated Show resolved Hide resolved

build-image/Dockerfile Outdated Show resolved Hide resolved

SkymanOne added 3 commits July 12, 2023 15:28

Merge branch 'master' into gn/verifiable-build

517b4bf

apply Andrews suggestions

b6b66e9

proper build step logs

590b5b8

SkymanOne requested a review from ascjones July 12, 2023 16:51

fix generation of ABI

d515504

ascjones reviewed Jul 12, 2023

View reviewed changes

crates/build/src/docker.rs Outdated Show resolved Hide resolved

crates/build/src/docker.rs Show resolved Hide resolved

hash the cmd, not entrypoint for digest

44958e3

ascjones approved these changes Jul 12, 2023

View reviewed changes

sergejparity approved these changes Jul 13, 2023

View reviewed changes

SkymanOne added 2 commits July 13, 2023 15:11

fix args parsing and add docs

ad1ddc6

clippy fix

6e02958

SkymanOne mentioned this pull request Jul 13, 2023

Follow-ups for the Verifiable Builds - #1148 #1214

Open

SkymanOne merged commit db24299 into master Jul 13, 2023
10 checks passed

SkymanOne deleted the gn/verifiable-build branch July 13, 2023 16:02

sourabhniyogi mentioned this pull request Jul 14, 2023

Standardised verifiable builds Brushfam/openbrush-contracts#110

Open

This was referenced Jul 20, 2023

Add information for reproducible builds to metadata #525

Closed

Release v3.1.0 #1237

Merged

ascjones mentioned this pull request Jul 27, 2023

Release v4.0.0-alpha #1243

Merged

SkymanOne mentioned this pull request Aug 10, 2023

add build image release workflow #1267

Merged

sourabhniyogi mentioned this pull request Sep 20, 2023

ChainIDE + Polkaholic.io WASM Contract Explorer Integration - Milestone 1 delivery use-inkubator/Grant-Milestone-Delivery#8

Merged

smiasojed mentioned this pull request Mar 4, 2024

Release v4.0.0 #1518

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standardised verifiable builds #1148

Standardised verifiable builds #1148

SkymanOne commented Jun 6, 2023 •

edited

Loading

SkymanOne commented Jun 7, 2023

ascjones left a comment

SkymanOne commented Jun 8, 2023 •

edited

Loading

deuszx commented Jun 9, 2023

ascjones left a comment

SkymanOne commented Jul 13, 2023 •

edited

Loading

sourabhniyogi commented Jul 13, 2023

SkymanOne commented Jul 13, 2023 •

edited

Loading

sourabhniyogi commented Jul 13, 2023 •

edited

Loading

SkymanOne commented Jul 14, 2023

wufengtao1 commented Jul 14, 2023 •

edited

Loading

SkymanOne commented Jul 14, 2023

wufengtao1 commented Jul 18, 2023

Standardised verifiable builds #1148

Standardised verifiable builds #1148

Conversation

SkymanOne commented Jun 6, 2023 • edited Loading

Tasks

Some extra goals I think worth considering

SkymanOne commented Jun 7, 2023

ascjones left a comment

Choose a reason for hiding this comment

SkymanOne commented Jun 8, 2023 • edited Loading

deuszx commented Jun 9, 2023

ascjones left a comment

Choose a reason for hiding this comment

SkymanOne commented Jul 13, 2023 • edited Loading

sourabhniyogi commented Jul 13, 2023

SkymanOne commented Jul 13, 2023 • edited Loading

sourabhniyogi commented Jul 13, 2023 • edited Loading

SkymanOne commented Jul 14, 2023

wufengtao1 commented Jul 14, 2023 • edited Loading

SkymanOne commented Jul 14, 2023

wufengtao1 commented Jul 18, 2023

SkymanOne commented Jun 6, 2023 •

edited

Loading

SkymanOne commented Jun 8, 2023 •

edited

Loading

SkymanOne commented Jul 13, 2023 •

edited

Loading

SkymanOne commented Jul 13, 2023 •

edited

Loading

sourabhniyogi commented Jul 13, 2023 •

edited

Loading

wufengtao1 commented Jul 14, 2023 •

edited

Loading