Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Upgrade RTS Dependencies #4677

Merged
merged 21 commits into from
Nov 29, 2024
Merged

chore: Upgrade RTS Dependencies #4677

merged 21 commits into from
Nov 29, 2024

Conversation

luc-blaeser
Copy link
Contributor

@luc-blaeser luc-blaeser commented Aug 29, 2024

Upgrading Motoko RTS dependencies:

  • LLVM 18.1.8
  • Rust 1.82.0

Special aspects:

  • Support new dylink.0 instead of dylink.
  • Support GOT.mem for RTS in the Motoko linker.
  • Adjusting the RTS code in line with the latest Rust features/rules.
  • No longer needing emscripten: Use custom Rust targets .
  • Get rid of emscripten by using custom Rust targets for generating PIC-relocatable Wasm.
  • Get rid of musl by using Rust builtins instead.
  • Float formatting changes due to removal of musl: The output format of NaN and infinity has changed. Moreover, one particular output format (hexadecimal float) is no longer supported. Updating the documentation and base library is needed.
  • Optimization: Use bulk-memory operations because memcpy of Rust is slower than musl.

Note:

  • If your nix-shell crashes with a segmentation fault with this PR, you need to upgrade nix. For this, you need to manually uninstall and re-install nix.

@luc-blaeser luc-blaeser self-assigned this Aug 29, 2024
@luc-blaeser luc-blaeser added the dependencies Pull requests that update a dependency file label Aug 29, 2024
Copy link

github-actions bot commented Aug 29, 2024

Comparing from aebc110 to bd845da:
In terms of gas, 3 tests regressed, 2 tests improved and the mean change is -0.0%.
In terms of size, 5 tests improved and the mean change is -1.6%.

@luc-blaeser luc-blaeser requested a review from ggreif August 29, 2024 12:58
@luc-blaeser luc-blaeser requested a review from crusso September 9, 2024 08:19
rts/Makefile Outdated Show resolved Hide resolved
luc-blaeser and others added 4 commits October 8, 2024 14:44
* Patch Rust for shared Wasm libraries

* Get rid of `musl`

* Further simplify Wasm linking

* Update benchmark results

* No longer need `wasm-ld` patch

* Update test result

* Add distinction for Linux Ocaml build

* Revert "No longer need `wasm-ld` patch"

This reverts commit 3d0f34e.

* Revert "Update test result"

This reverts commit 8fc5c08.

* Use patched wasm-ld during Rust build

* Table offset must be at least 1 for Rust RTS

* Adjust linker test result

* Use bulk memory operations

* Do not need `wasm-lld` patch with shared targets

* Adjust expected test result

* Adjust nix build script

* Fix nix build script
@luc-blaeser
Copy link
Contributor Author

Size compression is no longer needed for the specific Rust target backends.
Instead, we can again optimize for speed:

This is the latest perf diff on CI (locally repeated):

Comparing from b7fa63936cf708952deae0dd0b0429faa1baa119 to a10ab53a1ec3e5e48dbce36d6c7dd8547503c85c:
In terms of gas, 3 tests regressed, 2 tests improved and the mean change is -0.0%.
In terms of size, 5 tests improved and the mean change is -1.6%.

@luc-blaeser
Copy link
Contributor Author

According to the GC benchmark, it seems that this upgrade/cleanup even improves performance:

Mode Before Now Improvement
Classical (copying GC) 2.33e10 2.29e10 1.7%
Classical (incremental GC) 2.39e10 2.35e10 1.7%
EOP 2.54e0 2.50e10 1.6%

@luc-blaeser luc-blaeser requested a review from crusso October 9, 2024 16:17
nix/sources.json Outdated Show resolved Hide resolved
nix/default.nix Outdated Show resolved Hide resolved
default.nix Show resolved Hide resolved
ggreif
ggreif previously approved these changes Nov 25, 2024
Copy link
Contributor

@ggreif ggreif left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Heroic work on the dylink0 stuff. Being stuck on clang-13 gave me nightmares...
Nice to see the cycle improvements too. Getting rid of musl means one less wheel to maintain.

Good work! Added a few Qs...

crusso
crusso previously approved these changes Nov 26, 2024
Copy link
Contributor

@crusso crusso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! Not sure I fully understand the intricacies so 🤞

nix/sources.json Outdated Show resolved Hide resolved
rts/motoko-rts/src/allocator.rs Show resolved Hide resolved
src/linking/linkModule.ml Show resolved Hide resolved
src/linking/linkModule.ml Show resolved Hide resolved
@luc-blaeser luc-blaeser dismissed stale reviews from crusso and ggreif via c7422ff November 29, 2024 07:12
crusso added a commit to dfinity/motoko-base that referenced this pull request Nov 29, 2024
Adjusting the base library for latest Motoko RTS changes:
* Also test enhanced orthogonal persistence with Wasm Memory 64.
* Be prepared for the RTS dependency upgrade
dfinity/motoko#4677

Changes:
The original Musl float formatter has been replaced by the Rust
implementation with some subtle format changes:
- Numbers are displayed with the actually defined precision, no longer
truncating trailing zeros and potentially revealing existing numeric
errors.
- The exponent is printed with an explicit sign and multiple digits.
- NaN is formatted differently, now `NaN`, while the NaN sign bit is
omitted.
- The hexadecimal float formatter is no longer supported
(`Float.format(#hex)`). This is probably acceptable as it is rarely
used.

Once we have dfinity/motoko#4677 merged in
Motoko `master`, then we can remove the backwards compatibility of float
formatting in these tests. Backwards compatibility is here needed to
make `motoko-base` tests pass with the old Motoko compiler version.

---------

Co-authored-by: Claudio Russo <claudio@dfinity.org>
crusso
crusso previously approved these changes Nov 29, 2024
Copy link
Contributor

@crusso crusso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@luc-blaeser
Copy link
Contributor Author

Thank you for reviewing, Claudio and Gabor!

@luc-blaeser luc-blaeser added the automerge-squash When ready, merge (using squash) label Nov 29, 2024
@crusso crusso self-requested a review November 29, 2024 11:28
crusso
crusso previously approved these changes Nov 29, 2024
nix/default.nix Outdated Show resolved Hide resolved
revert default.nix change to adtgen
Copy link
Contributor

@ggreif ggreif left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mergify mergify bot merged commit bdb1d24 into master Nov 29, 2024
10 checks passed
@mergify mergify bot deleted the luc/upgrade-rts-dependencies branch November 29, 2024 17:14
@mergify mergify bot removed the automerge-squash When ready, merge (using squash) label Nov 29, 2024
ggreif pushed a commit to dfinity/motoko-base that referenced this pull request Dec 4, 2024
Adjusting the base library for latest Motoko RTS changes:
* Also test enhanced orthogonal persistence with Wasm Memory 64.
* Be prepared for the RTS dependency upgrade
dfinity/motoko#4677

Changes:
The original Musl float formatter has been replaced by the Rust
implementation with some subtle format changes:
- Numbers are displayed with the actually defined precision, no longer
truncating trailing zeros and potentially revealing existing numeric
errors.
- The exponent is printed with an explicit sign and multiple digits.
- NaN is formatted differently, now `NaN`, while the NaN sign bit is
omitted.
- The hexadecimal float formatter is no longer supported
(`Float.format(#hex)`). This is probably acceptable as it is rarely
used.

Once we have dfinity/motoko#4677 merged in
Motoko `master`, then we can remove the backwards compatibility of float
formatting in these tests. Backwards compatibility is here needed to
make `motoko-base` tests pass with the old Motoko compiler version.

---------

Co-authored-by: Claudio Russo <claudio@dfinity.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants