Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

disable string table compression on illumos in development #1129

Merged
merged 1 commit into from
Jun 15, 2022

Conversation

davepacheco
Copy link
Collaborator

See #1122. This change configures Cargo to pass -znostrcompress when building on illumos. This flag is documented with:

String tables are compressed by the link-editor by removing duplicate entries and tail substrings. This compression can significantly reduce the size of any string tables. A compressed .dynstr table can produce a smaller text segment and hence reduce runtime paging activity. Because of these benefits, string table compression is enabled by default.
Linking objects that contribute a very large number of symbols may increase the link-edit time due to the string table compression. To avoid this cost during development use the link-editors -z nocompstrtab option.

As I read this, the default behavior uses more link time to improve runtime performance over the long term. This doesn't feel like a great tradeoff for development. Using -znocompstrtab makes a notable improvement on my incremental rebuild time. I'll post some numbers. @rmustacc, I'd love a +1 from you that I'm not missing something important about the consequences of using this flag in development.

There are some caveats:

@jmpesp
Copy link
Contributor

jmpesp commented May 27, 2022

I set rustflags in order to reduce link time on my linux workstation:

[build]
rustflags = ["-C", "link-arg=-fuse-ld=lld"]

right now this is under [build] but I can set it for x86_64-unknown-linux-gnu instead. Ship it!

@davepacheco
Copy link
Collaborator Author

As a check that this at least worked, I ran:

while sleep 1; do pids="$(pgrep -x ld)" && pargs $pids && break; done

during a build, and caught this ld invocation:

$ while sleep 1; do pids="$(pgrep -x ld)" && pargs $pids && break; done5603:   /bin/ld -Y P,/usr/gcc/10/lib/amd64:/lib/amd64:/usr/lib/amd64 -R /usr/gcc/10/lib
argv[0]: /bin/ld
argv[1]: -Y
argv[2]: P,/usr/gcc/10/lib/amd64
argv[3]: -R
argv[4]: /usr/gcc/10/lib/amd64
argv[5]: -L
argv[6]: /usr/gcc/10/lib/amd64
argv[7]: -Qy
argv[8]: -o
argv[9]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47
argv[10]: /usr/lib/amd64/crt1.o
argv[11]: /opt/gcc-10/lib/gcc/x86_64-pc-solaris2.11/10.3.0/crtp.o
argv[12]: /usr/lib/amd64/crti.o
argv[13]: /usr/lib/amd64/values-Xc.o
argv[14]: /usr/lib/amd64/values-xpg6.o
argv[15]: /opt/gcc-10/lib/gcc/x86_64-pc-solaris2.11/10.3.0/crtbegin.o
argv[16]: -L/home/dap/omicron/target/debug/deps
argv[17]: -L/home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib
argv[18]: -L/home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib
argv[19]: -L/opt/gcc-10/lib/gcc/x86_64-pc-solaris2.11/10.3.0
argv[20]: -L/opt/gcc-10/lib/gcc/x86_64-pc-solaris2.11/10.3.0/../../../amd64
argv[21]: -L/opt/gcc-10/lib/gcc/x86_64-pc-solaris2.11/10.3.0/../../..
argv[22]: /dangerzone/omicron_tmp/rustcGzmhbt/symbols.o
argv[23]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.0.rcgu.o
argv[24]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.1.rcgu.o
argv[25]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.10.rcgu.o
argv[26]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.11.rcgu.o
argv[27]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.12.rcgu.o
argv[28]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.13.rcgu.o
argv[29]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.14.rcgu.o
argv[30]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.15.rcgu.o
argv[31]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.2.rcgu.o
argv[32]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.3.rcgu.o
argv[33]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.4.rcgu.o
argv[34]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.5.rcgu.o
argv[35]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.6.rcgu.o
argv[36]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.7.rcgu.o
argv[37]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.8.rcgu.o
argv[38]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.build_script_build.24e698ff-cgu.9.rcgu.o
argv[39]: /home/dap/omicron/target/debug/build/serde_derive-0aea54b747daca47/build_script_build-0aea54b747daca47.3cw4tnbzfnc8ivfu.rcgu.o
argv[40]: -z
argv[41]: ignore
argv[42]: --start-group
argv[43]: -Bstatic
argv[44]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libstd-f2001cfa3748d4df.rlib
argv[45]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libpanic_unwind-6f31440dc375857e.rlib
argv[46]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libobject-bb7a6589b0664a20.rlib
argv[47]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libmemchr-ea407fcb6be59405.rlib
argv[48]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libaddr2line-ae38d43bf4b09454.rlib
argv[49]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libgimli-e55bf8816e492d27.rlib
argv[50]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/librustc_demangle-7b73127e4265cfbf.rlib
argv[51]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libstd_detect-5d49d6cdd634c003.rlib
argv[52]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libhashbrown-acdbf9418f5ed2e6.rlib
argv[53]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libminiz_oxide-ebb320bb1f835dda.rlib
argv[54]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libadler-d7889c4cdf1d8b06.rlib
argv[55]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/librustc_std_workspace_alloc-5ce31d4c9122a5d4.rlib
argv[56]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libunwind-dc17fab03a81373c.rlib
argv[57]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libcfg_if-38eebd1e09ca6ed7.rlib
argv[58]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/liblibc-3c261ead1b95074a.rlib
argv[59]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/liballoc-f2d1f6c2f54bcd69.rlib
argv[60]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/librustc_std_workspace_core-5a6b7292ae0792e1.rlib
argv[61]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libcore-bb36fb4d6ea14041.rlib
argv[62]: --end-group
argv[63]: /home/dap/.rustup/toolchains/nightly-2022-04-27-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib/libcompiler_builtins-0f2fb3af18bde56a.rlib
argv[64]: -Bdynamic
argv[65]: -lsocket
argv[66]: -lposix4
argv[67]: -lpthread
argv[68]: -lresolv
argv[69]: -lnsl
argv[70]: -lumem
argv[71]: -lgcc_s
argv[72]: -lm
argv[73]: -lrt
argv[74]: -lpthread
argv[75]: -lsendfile
argv[76]: -llgrp
argv[77]: -lc
argv[78]: -lssp
argv[79]: -znocompstrtab
argv[80]: /opt/gcc-10/lib/gcc/x86_64-pc-solaris2.11/10.3.0/crtend.o
argv[81]: /usr/lib/amd64/crtn.o

argv[79] shows we're getting the flag all the way to ld.

@davepacheco
Copy link
Collaborator Author

With this change, the build took 7m13s:

dap@ivanova omicron $ cargo clean
dap@ivanova omicron $ time cargo test
...
    Finished test [unoptimized + debuginfo] target(s) in 7m 13s

The test part was interrupted by #1130. A subsequent cargo test did no build and took 2m27s:

$ time cargo test
    Finished test [unoptimized + debuginfo] target(s) in 0.48s
     Running unittests src/lib.rs (target/debug/deps/authz_macros-bc4c3415e71f60e1)
...
   Doc-tests sp-sim

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


real    2m27.071s
user    7m59.655s
sys     2m48.890s

After that, an incremental build took 1m20s:

$ time cargo test -- not-a-real-test
   Compiling omicron-nexus v0.1.0 (/home/dap/omicron/nexus)
   Compiling nexus-test-utils v0.1.0 (/home/dap/omicron/nexus/test-utils)
    Finished test [unoptimized + debuginfo] target(s) in 1m 20s
...
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


real    1m41.348s
user    2m2.024s
sys     0m59.703s

(note: using the reported build time there rather than the time output because it took 20s more just to scan all the tests!)

Without this change (from the tip of "main", 236f2ac), the full build and test took 10m23s:

$ cargo clean
$ time cargo test
...
   Compiling omicron-deploy v0.1.0 (/home/dap/omicron/deploy)
    Finished test [unoptimized + debuginfo] target(s) in 7m 52s
     Running unittests src/lib.rs (target/debug/deps/authz_macros-bc4c3415e71f60e1)
...
running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


real    10m23.081s
user    68m43.292s
sys     12m53.390s

and the incremental build took 1m46s:

$ touch nexus/src/app/mod.rs 
dap@ivanova omicron $ time cargo test -- not-a-real-test
   Compiling omicron-nexus v0.1.0 (/home/dap/omicron/nexus)
   Compiling nexus-test-utils v0.1.0 (/home/dap/omicron/nexus/test-utils)
    Finished test [unoptimized + debuginfo] target(s) in 1m 46s
     Running unittests src/lib.rs (target/debug/deps/authz_macros-bc4c3415e71f60e1)
...
real    2m7.475s
user    3m15.884s
sys     0m55.370s

(using the reported build time for the same reason as above).

So this change shaved about 43 seconds off the full build time (7%) and 26 seconds off the incremental build (20%).

@davepacheco davepacheco marked this pull request as ready for review May 27, 2022 20:50
@rmustacc
Copy link

The only real tradeoff I can think of right now would be resulting binary size and any knock on effects from that. But I anticipate that's not a big concern here.

@davepacheco davepacheco merged commit 84390f8 into main Jun 15, 2022
@davepacheco davepacheco deleted the link-nostrcompress branch June 15, 2022 21:08
leftwo pushed a commit that referenced this pull request Feb 9, 2024
Crucible changes:
Remove unused fields in IOop (#1149)
New downstairs clone subcommand. (#1129)
Simplify the do_work_task loop (#1150)
Move `Guest` stuff into a module (#1125)
Bump nix to 0.27.1 and use new safer Fd APIs (#1110)
Move `FramedWrite` work to a separate task (#1145)
Use fewer borrows in ExtentInner API (#1147)
Update Rust crate reedline to 0.28.0 (#1141)
Update Rust crate tokio to 1.36 (#1143)
Update Rust crate slog-bunyan to 2.5.0 (#1139)
Update Rust crate rayon to 1.8.1 (#1138)
Update Rust crate itertools to 0.12.1 (#1137)
Update Rust crate byte-unit to 5.1.4 (#1136)
Update Rust crate base64 to 0.21.7 (#1135)
Update Rust crate async-trait to 0.1.77 (#1134)
Discard deferred msgs (#1131)
Minor Downstairs cleanup (#1127)
Update test_fail_live_repair to support pstop (#1128)
Ignore client messages after stopping the IO task (#1126)
Move client IO task into a struct (#1124)
Bump Rust to 1.75 and fix new Clippy lints (#1123)

Propolis changes:
PHD: convert to async (#633)
PHD: assume specialized Windows images (#636)
propolis-standalone-config needn't be a crate
standalone: Use tar for snapshot/restore
phd: use latest "lab-2.0-opte" target, not a specific version (#637)
PHD: add tests for migration of running processes (#623)
PHD: fix `cargo xtask phd` tidy not doing anything (#630)
PHD: add documentation for `cargo xtask phd` (#629)
standalone: improve virtual device creation errors (#632)
phd: add Windows Server 2019 guest adapter (#627)
PHD: add `cargo xtask phd` to make using PHD nicer (#619)
leftwo added a commit that referenced this pull request Feb 9, 2024
Crucible changes:
Remove unused fields in IOop (#1149)
New downstairs clone subcommand. (#1129)
Simplify the do_work_task loop (#1150)
Move `Guest` stuff into a module (#1125)
Bump nix to 0.27.1 and use new safer Fd APIs (#1110) Move `FramedWrite`
work to a separate task (#1145) Use fewer borrows in ExtentInner API
(#1147)
Update Rust crate reedline to 0.28.0 (#1141)
Update Rust crate tokio to 1.36 (#1143)
Update Rust crate slog-bunyan to 2.5.0 (#1139)
Update Rust crate rayon to 1.8.1 (#1138)
Update Rust crate itertools to 0.12.1 (#1137)
Update Rust crate byte-unit to 5.1.4 (#1136)
Update Rust crate base64 to 0.21.7 (#1135)
Update Rust crate async-trait to 0.1.77 (#1134)
Discard deferred msgs (#1131)
Minor Downstairs cleanup (#1127)
Update test_fail_live_repair to support pstop (#1128) Ignore client
messages after stopping the IO task (#1126) Move client IO task into a
struct (#1124)
Bump Rust to 1.75 and fix new Clippy lints (#1123)

Propolis changes:
PHD: convert to async (#633)
PHD: assume specialized Windows images (#636)
propolis-standalone-config needn't be a crate
standalone: Use tar for snapshot/restore
phd: use latest "lab-2.0-opte" target, not a specific version (#637)
PHD: add tests for migration of running processes (#623) PHD: fix `cargo
xtask phd` tidy not doing anything (#630) PHD: add documentation for
`cargo xtask phd` (#629) standalone: improve virtual device creation
errors (#632) phd: add Windows Server 2019 guest adapter (#627)
PHD: add `cargo xtask phd` to make using PHD nicer (#619)

Co-authored-by: Alan Hanson <alan@oxide.computer>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants