feat: dynamic wasm_url and wasm_hash

[lwshang]

Description

Implement SDK-1211

wasm_url and wasm_hash might need to be calculated dynamically during canister build.

wasm_url

Beyond setting wasm_url directly, you can generate it into a file with configuration in dfx.json like:

{
  "pullable" : {
    "dynamic_wasm_url" : {
      "generate" : "<commands to generate wasm_url file>",
      "path" : "<path/to/wasm_url_file>"
    }
  }
}

You can only set one of wasm_url and dynamic_wasm_url.

wasm_hash

Service providers may want consumers to download a different wasm module than the on-chain canister.

In such case, providers need to include a wasm_hash field in the dfx/pullable metadata of the on-chain pullable canister.

There are three ways to specify wasm_hash in dfx.json. You can choose the most suitable one for your use.

1. Set wasm_hash statically:

{
  "pullable" : {
    "wasm_hash" : "<hex-encoded SHA256 hash of the wasm module to be downloaded>"
  }
}

2. Set wasm_hash_file, then dfx read its content to set wasm_hash:

{
  "pullable" : {
    "wasm_hash_file" : "<path/to/wasm_hash_file>"
  }
}

3. Set custom_wasm which run generate commands and use the custom wasm module to calculate wasm_hash:

{
  "pullable" : {
    "custom_wasm" : {
      "generate" : "<commands to generate custom wasm file>",
      "path" : "<path/to/custom_wasm_file>"
    }
  }
}

dfx will do the same post-processing (optimize, metadata, gzip) to the custom wasm as to the canister to be deployed on-chain. wasm_hash will be calculated from the post-processed wasm.

How Has This Been Tested?

e2e test in deps.bash

Checklist:

• The title of this PR complies with Conventional Commits.
• I have edited the CHANGELOG accordingly.
• I have made corresponding changes to the documentation.

[frederikrothenberger]

Thanks @lwshang for working on this. However, there is still one case that is not clear to me how to solve: The issue of multiple build types.

Example:
II has 3 build types: dev, test and prod

Our dfx.json currently looks like this:

{
  "canisters": {
    "internet_identity": {
      "type": "custom",
      "candid": "src/internet_identity/internet_identity.did",
      "wasm": "internet_identity.wasm.gz",
      "build": "scripts/build",
      "shrink" : false
    }
  },
  ...
}

Depending on the environment variables you set, scripts/build builds a different build type.

I.e. we can run
II_FETCH_ROOT_KEY=1 II_DUMMY_CAPTCHA=1 II_INSECURE_REQUESTS=1 dfx deploy internet_identity to deploy the dev build locally. Or we can simply run dfx deploy internet_identity to deploy the prod build locally.

Now, to build the prod build correctly, embedding the dev build hash in the metadata we would have to do this (if I understand correctly):

{
  "canisters": {
    "internet_identity": {
      "type": "custom",
      "candid": "src/internet_identity/internet_identity.did",
      "wasm": "internet_identity.wasm.gz",
      "build": "scripts/build",
      "shrink" : false,
      "pullable": {
        "wasm_url": "http://example.com/a.wasm",
        "custom_wasm" : {
          "generate" : "II_FETCH_ROOT_KEY=1 II_DUMMY_CAPTCHA=1 II_INSECURE_REQUESTS=1 scripts/build",
          "path" : "./internet_identity_dev.wasm.gz"
        },
        "dependencies": [],
        "init_guide": "..."
      }
    }
  },
  ...
}

However, now dfx deploy internet_identity cannot deploy the dev build anymore (as building internet_identity now implies always building 2 things, the wasm to be hashed and the actual wasm to embed the hash in.

What we would like to have is this:

• prod build pulls dev build --> embeds dev build hash
• dev build pulls dev build --> no hash
• test build pulls test build --> no hash

In order to do that, we need some way of dynamically opting-out of including a hash or we need a proper representation of build flavours of the same(!) canister in dfx.json.

[lwshang]

@frederikrothenberger Thanks for the feedback! It is my goal to make this feature works for internet-identity.

The dfx.json in your comment is exactly what I imagined. With such dfx.json configuration, when running dfx build or dfx deploy, it will always build a wasm to be hashed and attach the wasm_hash to the canister wasm to be deployed.

While it is still possible to deploy the dev build. When you run II_FETCH_ROOT_KEY=1 II_DUMMY_CAPTCHA=1 II_INSECURE_REQUESTS=1 dfx deploy internet_identity, it will build the dev flavor twice that has different dfx/pullable metadata. The one with valid wasm_hash will be deployed to local replica.

It does no harm to embed wasm_hash in dev and test build.

If the main concern is the extra build time, we can optimize that by altering the generate script.
For example, you can set empty canisters->pullable->custom_wasm->generate and explicitly run II_FETCH_ROOT_KEY=1 II_DUMMY_CAPTCHA=1 II_INSECURE_REQUESTS=1 scripts/build to build the dev flavor only when necessary. Then dfx build or dfx deploy won't incur the extra build time.

[frederikrothenberger]

While it is still possible to deploy the dev build. When you run II_FETCH_ROOT_KEY=1 II_DUMMY_CAPTCHA=1 II_INSECURE_REQUESTS=1 dfx deploy internet_identity, it will build the dev flavor twice that has different dfx/pullable metadata. The one with valid wasm_hash will be deployed to local replica.

This is really confusing. Now there are two different dev builds: One with the hash embedded of the other, but both pullable. One self-referencing and the other pointing to the non-hash-embedded one...

For example, you can set empty canisters->pullable->custom_wasm->generate and explicitly run II_FETCH_ROOT_KEY=1 II_DUMMY_CAPTCHA=1 II_INSECURE_REQUESTS=1 scripts/build to build the dev flavor only when necessary.

This requires modifying the dfx.json to switch build profiles. 🙁

Just give us the option to have the generate script create an empty file to opt out of embedding anything. This seems like a cleaner solution without incurring compilation work, changes to dfx.json and multiple variants of the dev build...

[lwshang]

#3510 instead