Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: rebuild only necessary canisters #3710

Open
wants to merge 388 commits into
base: master
Choose a base branch
from

Conversation

vporton
Copy link
Contributor

@vporton vporton commented Apr 14, 2024

Description

DFX doesn't compile canisters for which all dependencies are elder than the .wasm file. This results in big compilation speedups.

The PR also adds tracing and comments intended for future improvement of compilation speed.

This PR is a WIP. There is potential for further speedup and there is missing logging. More testing is necessary.

How Has This Been Tested?

I run it before the .wasm file has been created and after it with a significant speedup. Also dfx build -vv correctly does not output any moc compilation logs, when invoking dfx build for the second time.

Later I did bats make_like.bash, add_dependency.bash, broken_canister_dep.bash, and dotenv.bash automated testing.

Checklist:

  • The title of this PR complies with Conventional Commits.
  • I have edited the CHANGELOG accordingly.
  • I have made corresponding changes to the documentation.

@vporton
Copy link
Contributor Author

vporton commented Apr 14, 2024

dfx deploy of my current DFX version running in https://github.com/vporton/zondirectory2 repo causes an infinite loop (but dfx build --all doesn't).

@vporton
Copy link
Contributor Author

vporton commented Apr 16, 2024

The last commit (79faa772dd3b8a7683ed9670a1447ecd0b2893ba) seems working!

Also note that unlike the original (master) code, my code does recompile dependent canisters (canister:* dependencies) when a dependency changed.

The code needs refactoring and more serious testing.

@vporton
Copy link
Contributor Author

vporton commented Apr 16, 2024

Testing shows a bug: When specifying a dependency to build: dfx build dependency, the dependent canister may recompile, too.

@vporton vporton force-pushed the build-only-necessary branch from 4be4fa8 to 030f9af Compare April 16, 2024 15:58
@vporton
Copy link
Contributor Author

vporton commented Apr 16, 2024

The latest version commit 030f9af902515edd70d60acde5d2cb547c200b40 works as expected on simple (manual) tests (with more tests than the past time).

You may try to use it for your home projects, but don't try it for mission-critical tasks or if you fear to be lost in wrong recompilations, because errors are still possible.

@vporton
Copy link
Contributor Author

vporton commented Apr 16, 2024

Fixed crash on dfx build --all (creates a new bug, about compiling non-Motoko canister: dependencies).

@vporton vporton requested a review from a team as a code owner May 31, 2024 13:49
@vporton
Copy link
Contributor Author

vporton commented May 31, 2024

This MR has been "battle-tested" a little. It worked as expected in developing https://github.com/vporton/zondirectory2 and in testing of https://github.com/vporton/joining-proxy-rust without any visible errors.

@vporton
Copy link
Contributor Author

vporton commented Jun 9, 2024

There has been found yet a bug:

Compiling vporton/zondirectory2@b145a4c it was output:

...

The command '"moc-wrapper" "src/backend/personhood.mo" "-o" "/home/porton/Projects/zondirectory/.dfx/local/canisters/personhood/personhood.wasm" "-c" "--debug" "--idl" "--stable-types" "--public-metadata" "candid:service" "--public-metadata" "candid:args" "--actor-idl" "/home/porton/Projects/zondirectory/.dfx/local/canisters/idl/" "--actor-alias" "CanDBIndex" "bkyz2-fmaaa-aaaaa-qaaaq-cai" "--actor-alias" "ic_eth" "avqkn-guaaa-aaaaa-qaaea-cai" "--package" "base" ".mops/base@0.11.1/src" "--package" "candb" ".mops/_github/candb#no-blob/src" "--package" "nacdb" ".mops/nacdb@0.13.0/src" "--package" "stable-rbtree" ".mops/stable-rbtree@1.0.0/src" "--package" "stable-buffer" ".mops/_github/stable-buffer#v0.2.0/src" "--package" "prng" ".mops/prng@0.0.3/src" "--package" "stableheapbtreemap" ".mops/stableheapbtreemap@1.3.0/src" "--package" "sha2" ".mops/sha2@0.0.4/src" "--package" "stablebuffer" ".mops/_github/stablebuffer#v0.2.0/src" "--package" "StableBuffer" ".mops/_github/StableBuffer#v0.2.0/src" "--package" "icrc1" ".mops/icrc1@0.0.2/src" "--package" "array" ".mops/_github/array#main/src" "--package" "StableTrieMap" ".mops/_github/StableTrieMap#main/src" "--package" "itertools" ".mops/itertools@0.1.2/src" "--package" "base-0.7.3" ".mops/_github/base-0.7.3#aafcdee0c8328087aeed506e64aa2ff4ed329b47/src" "--package" "xtendedNumbers" ".mops/_github/xtendedNumbers#v1.1.0/src" "--package" "motoko-lib" ".mops/_github/motoko-lib#0.7/src" "--package" "map7" ".mops/_github/map7#v7.0.0/src" "--package" "sha" ".mops/_github/sha#master/src" "--package" "matchers" ".mops/_github/matchers#v1.3.0/src" "--package" "map" ".mops/map@9.0.1/src" "--package" "btree" ".mops/_github/btree#v0.3.3/src" "--package" "CanDBMulti" ".mops/_github/CanDBMulti#v0.10.0@86d00711216349ba4879370d6313d44a26d16f61/src" "--package" "xtended-numbers" ".mops/xtended-numbers@0.2.1/src" "--package" "json.mo" ".mops/json.mo@0.1.2/src" "--package" "parser-combinators" ".mops/_github/parser-combinators#v0.1.2/src" "--package" "nacdb-reorder" ".mops/nacdb-reorder@3.0.0/src" "--package" "passport-client" ".mops/passport-client@0.4.6/src" "--package" "sequence" ".mops/_github/sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src" "--package" "stable-hash-map" ".mops/_github/stable-hash-map#v0.2.1@eb7edf4127233f2b1f3732ede7e2c55827ac6886/src" "--package" "encoding" ".mops/_github/encoding#v0.3.2@8e0fe1d8f5c2d284e77d719703c42e0e271839b1/src" "--package" "candy" ".mops/_github/candy#v0.3.0@907a4e7363aac6c6a4e114ebc73e3d3f21e138af/src" "--package" "stablebuffer_1_3_0" ".mops/_github/stablebuffer_1_3_0#v1.3.0@acdde6bb5b939227997cebdbb8919d2e6da8691c/src" "--package" "map9" ".mops/_github/map9#v9.0.1@10b68f6ea8df5e72dfa4c07a50c8bb60a916c233/src" "--package" "candb-multi" ".mops/candb-multi@0.11.0/src" "--package" "test" ".mops/test@1.2.0/src" "--package" "date.mo" ".mops/_github/date.mo#master@cfe51ad7d8a576d53e701136d4234233b94970eb/src" "--package" "testing" ".mops/_github/testing#main/src" "--package" "fmt" ".mops/_github/fmt#v0.1.0/src" "--package" "motoko-sequence" ".mops/_github/motoko-sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src" "-fshared-code" "--actor-idl" ".dfx/local/lsp"' failed with exit status 'exit status: 1'. Stdout:

Stderr: src/backend/personhood.mo:5.1-5.40: import error [M0009], file ".dfx/local/lsp/bkyz2-fmaaa-aaaaa-qaaaq-cai.did" does not exist

src/backend/personhood.mo:5.1-5.40 is about CanDBIndex (bkyz2-fmaaa-aaaaa-qaaaq-cai).

@vporton
Copy link
Contributor Author

vporton commented Jun 9, 2024

I've removed "that's weird" from my comment, because it has been written by me in error.

@vporton
Copy link
Contributor Author

vporton commented Jun 9, 2024

Note that the above error was caused by dfx start --background --clean && dfx deploy frontend.

@vporton
Copy link
Contributor Author

vporton commented Jun 9, 2024

There has been found yet a bug:

I run dfx deploy frontend 15 times and wasn't able to reprise the above bug. Probably, I just installed a wrong version of dfx and there is no bug in this version. I will keep using it in practice.

@vporton
Copy link
Contributor Author

vporton commented Jun 16, 2024

I've fixed a serious bug.

@vporton
Copy link
Contributor Author

vporton commented Jun 25, 2024

Found bugs, while attempting deploying a real app to ic network:

  1. ic_eth (Rust) canister was decided to be compiled after a Motoko canister dependent on it.
  2. Trying to deploy a real app to ic network:
Caused by: The command '"moc-wrapper" "/home/porton/Projects/zondirectory/src/backend/personhood.mo" "-o" "/home/porton/Projects/zondirectory/.dfx/ic/canisters/personhood/personhood.wasm" "-c" "--debug" "--idl" "--stable-types" "--public-metadata" "candid:service" "--public-metadata" "candid:args" "--actor-idl" "/home/porton/Projects/zondirectory/.dfx/ic/canisters/idl/" "--actor-alias" "CanDBIndex" "n674v-zqaaa-aaaap-qbpza-cai" "--actor-alias" "ic_eth" "jrzee-liaaa-aaaap-qhnoa-cai" "--package" "base" ".mops/base@0.11.1/src" "--package" "candb" ".mops/_github/candb#no-blob/src" "--package" "nacdb" ".mops/nacdb@0.13.0/src" "--package" "stable-rbtree" ".mops/stable-rbtree@1.0.0/src" "--package" "stable-buffer" ".mops/_github/stable-buffer#v0.2.0/src" "--package" "prng" ".mops/prng@0.0.3/src" "--package" "stableheapbtreemap" ".mops/stableheapbtreemap@1.3.0/src" "--package" "sha2" ".mops/sha2@0.0.4/src" "--package" "stablebuffer" ".mops/_github/stablebuffer#v0.2.0/src" "--package" "StableBuffer" ".mops/_github/StableBuffer#v0.2.0/src" "--package" "icrc1" ".mops/icrc1@0.0.2/src" "--package" "array" ".mops/_github/array#main/src" "--package" "StableTrieMap" ".mops/_github/StableTrieMap#main/src" "--package" "itertools" ".mops/itertools@0.1.2/src" "--package" "base-0.7.3" ".mops/_github/base-0.7.3#aafcdee0c8328087aeed506e64aa2ff4ed329b47/src" "--package" "xtendedNumbers" ".mops/_github/xtendedNumbers#v1.1.0/src" "--package" "motoko-lib" ".mops/_github/motoko-lib#0.7/src" "--package" "map7" ".mops/_github/map7#v7.0.0/src" "--package" "sha" ".mops/_github/sha#master/src" "--package" "matchers" ".mops/_github/matchers#v1.3.0/src" "--package" "map" ".mops/map@9.0.1/src" "--package" "btree" ".mops/_github/btree#v0.3.3/src" "--package" "xtended-numbers" ".mops/xtended-numbers@0.2.1/src" "--package" "json.mo" ".mops/_github/json.mo#master@49612cf3865dc24bca7bb0e8b6fcf275d11a8cb2/src" "--package" "parser-combinators" ".mops/_github/parser-combinators#main/src" "--package" "nacdb-reorder" ".mops/nacdb-reorder@3.1.1/src" "--package" "cycles-simple" ".mops/cycles-simple@0.11.0/src" "--package" "stablehashmap" ".mops/stablehashmap@1.0.0/src" "--package" "candb-multi" ".mops/candb-multi@0.11.8/src" "--package" "stable-hash-map" ".mops/_github/stable-hash-map#v0.2.1@eb7edf4127233f2b1f3732ede7e2c55827ac6886/src" "--package" "encoding" ".mops/_github/encoding#v0.3.2@8e0fe1d8f5c2d284e77d719703c42e0e271839b1/src" "--package" "candy" ".mops/_github/candy#v0.3.0@907a4e7363aac6c6a4e114ebc73e3d3f21e138af/src" "--package" "stablebuffer_1_3_0" ".mops/_github/stablebuffer_1_3_0#v1.3.0@acdde6bb5b939227997cebdbb8919d2e6da8691c/src" "--package" "map9" ".mops/_github/map9#v9.0.1@10b68f6ea8df5e72dfa4c07a50c8bb60a916c233/src" "--package" "join-proxy-motoko" ".mops/_github/join-proxy-motoko#master@3582456b29cb81e0e304f7df2a3935da670725f6/src" "--package" "parser-combinators.mo" ".mops/_github/parser-combinators.mo#master@f87c9417ab145017fb1f83e19ec2fc45588dbd41/src" "--package" "passport-client" ".mops/passport-client@0.4.9/src" "--package" "sequence" ".mops/_github/sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src" "--package" "test" ".mops/test@1.2.0/src" "--package" "date.mo" ".mops/_github/date.mo#master@cfe51ad7d8a576d53e701136d4234233b94970eb/src" "--package" "testing" ".mops/_github/testing#main/src" "--package" "fmt" ".mops/_github/fmt#v0.1.0/src" "--package" "motoko-sequence" ".mops/_github/motoko-sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src"' failed with exit status 'exit status: 1'.
Stdout:

Stderr:
/home/porton/Projects/zondirectory/src/backend/personhood.mo:5.1-5.40: import error [M0009], file "/home/porton/Projects/zondirectory/.dfx/ic/canisters/idl/n674v-zqaaa-aaaap-qbpza-cai.did" does not exist
/home/porton/Projects/zondirectory/src/backend/personhood.mo:6.1-6.32: import error [M0009], file "/home/porton/Projects/zondirectory/.dfx/ic/canisters/idl/jrzee-liaaa-aaaap-qhnoa-cai.did" does not exist
/home/porton/Projects/zondirectory/src/backend/personhood.mo:9.1-9.56: import error [M0009], file "/home/porton/Projects/zondirectory/src/libs/configs/passport.config/lib.mo" does not exist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants