cdylib artifact size boost #45613

king6cong · 2017-10-29T10:15:23Z

Hello, I find there is a huge boost in artifact size of cdylib library files in recent versions of rust. In our case, the lib doubles its size. I also find similar situations in other projects.

Here are the steps to reproduce (OS: 64bit Mac OS X 10.12.4 16E195):

git clone https://github.com/king6cong/sysinfo
cd sysinfo

on nightly-2017-06-30

rustup default nightly-2017-06-30
cargo clean
cargo build --features="c-interface"
ls -lht target/debug/libsysinfo.dylib
nm -g target/debug/libsysinfo.dylib|less

latest nightly (rustc 1.23.0-nightly (`269cf50` 2017-10-28))

rustup default nightly
cargo clean
cargo build --features="c-interface"
ls -lht target/debug/libsysinfo.dylib
nm -g target/debug/libsysinfo.dylib|less

Comparing the output:

# nightly-2017-06-30
751K Oct 29 17:59 target/debug/libsysinfo.dylib
# latest nightly
2.2M Oct 29 17:52 target/debug/libsysinfo.dylib

and there are many rust specific symbols in the latest nightly generated lib.

Library size matters, especially on mobile devices such as iOS and Android, definitely eager to figure out the reason :)

The text was updated successfully, but these errors were encountered:

nagisa · 2017-10-29T10:46:39Z

target/debug

This is a debug build. I’m not surprised at all if it began putting more symbols in the files.

nagisa · 2017-10-29T10:55:58Z

cc @michaelwoerister is this fixed by one of your #[inline] instantiation fixes?

king6cong · 2017-10-29T11:36:01Z

@nagisa similar result with release version:

on nightly-2017-06-30

rustup default nightly-2017-06-30
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.dylib
nm -g target/release/libsysinfo.dylib|less

latest nightly (rustc 1.23.0-nightly (`269cf50` 2017-10-28))

rustup default nightly
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.dylib
nm -g target/release/libsysinfo.dylib|less

Comparing the release output:

260K Oct 29 17:59 target/release/libsysinfo.dylib
1.7M Oct 29 17:52 target/release/libsysinfo.dylib

still, many rust specific symbols in the latest nightly generated lib, while nightly-2017-06-30 works as expected.

Mark-Simulacrum · 2017-10-30T01:46:29Z

Nominating. Would be good to track this down, to at least diagnose the cause and add to the release notes. Probably not too high priority...

michaelwoerister · 2017-10-30T09:41:57Z

Thanks for the report, @king6cong! This looks indeed like a bug. We haven't changed anything in our compilation strategy the would require such a change.

is this fixed by one of your #[inline] instantiation fixes?

That might have a positive effect here, but since inline instantiation seems to have regressed already at the beginning of this year, something else must be the problem.

cc @alexcrichton

alexcrichton · 2017-10-30T17:17:50Z

I believe this is all debug information. On Linux tonight's nightly is 3.0MB and nightly-2017-06-30 is 2.4MB. After strip -g both are 1.4MB.

king6cong · 2017-10-31T03:03:42Z

@alexcrichton

I tried on Linux (x86_64 Linux 4.12.12-1-ARCH) and reproduced the problem:

on nightly-2017-06-30

rustup default nightly-2017-06-30
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.so
strip -g target/release/libsysinfo.so
ls -lht target/release/libsysinfo.so
nm -g target/release/libsysinfo.so|less

on latest nightly

rustup default nightly
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.so
strip -g target/release/libsysinfo.so
ls -lht target/release/libsysinfo.so
nm -g target/release/libsysinfo.so|less

Comparing the output on Linux:

# nightly-2017-06-30
# before strip
2.1M Oct 31 13:52 target/release/libsysinfo.so
# after strip
403K Oct 31 13:52 target/release/libsysinfo.so

# latest nightly
# before strip
3.0M Oct 31 13:54 target/release/libsysinfo.so
# after strip
1.4M Oct 31 13:54 target/release/libsysinfo.so

similar result for strip -x on macOS:

# nightly-2017-06-30
260K # before strip
152K # after strip
# latest nightly
1.7M # before strip
1.1M # after strip

And we can see many rust specific symbols in the striped lastest nightly version.

alexcrichton · 2017-10-31T03:51:31Z

Huh that's odd, I'm not sure what happened before... Now I can reproduce the results you're getting!

Somehow this is related to --crate-type rlib --crate-type cdylib (both at the same time). When I drop the rlib crate type then current nightly is as small than the older nightly.

alexcrichton · 2017-10-31T04:01:39Z

Bisection via nightlies points to ae98ebf...15aa15b (rust-nightly-2017-07-21 vs rust-nightly-2017-07-22) of which the most likely candidate is #43183.

@michaelwoerister I think the change here was that the set of exported symbols doesn't take the crate type as input, so we pessimistically export all symbols as if we were making an rlib, but apparently before we did a better job of internalizing things via the linker I guess?

@king6cong is it problematic to split the project into a rlib crate and a cdylib crate? (vs having both in one). Failing that we can try to fix this in rustc.

king6cong · 2017-10-31T07:52:32Z

@alexcrichton Great, that works!

# macOS
256K/148K # older nightly w/o strip
305K/194K # latest nightly w/o strip

# Linux
2.1M/403K # older nightly w/o strip
2.0M/393K # latest nightly w/o strip

For our project, the dylib size dropped from 23M to 9M, a huge difference!

michaelwoerister · 2017-10-31T10:08:55Z

This looks kind of wrong:

rust/src/librustc_trans/back/linker.rs

Lines 747 to 755 in 6713736

    
           fn exported_symbols(tcx: TyCtxt, crate_type: CrateType) -> Vec<String> { 
        
               let mut symbols = Vec::new(); 
        
               let export_threshold = symbol_export::threshold(tcx); 
        
               for &(ref name, _, level) in tcx.exported_symbols(LOCAL_CRATE).iter() { 
        
                   if level.is_below_threshold(export_threshold) { 
        
                       symbols.push(name.clone()); 
        
                   } 
        
               }

It does not take the crate type into account when computing the export threshold.

@alexcrichton

…bol-threshold, r=alexcrichton Take crate-type into account when generating symbol export lists (linker version scripts) r? @alexcrichton cc rust-lang#45613

michaelwoerister · 2017-11-02T20:25:27Z

#45650 should improve the situation here. @king6cong, could you give it another try with the latest nightly?

nikomatsakis · 2017-11-02T20:26:13Z

triage: P-medium

Discussed in compiler team. @michaelwoerister believes may be fixed, and in any case will pursue. Calling it P-medium because it only affects binary size and only if you use two crate-types at once.

king6cong · 2017-11-03T12:25:16Z

@michaelwoerister Tested in sysinfo and our project, both worked as expected, great!
Thanks 😄

rkarp · 2017-11-03T16:04:32Z

There is still a big size difference on windows-gnu when building DLLs with crate-type = ["cdylib"], and it wasn't introduced in the suspected 2017-07-22 nightly.

Latest nightly (nightly-2017-11-02-x86_64-pc-windows-gnu):

cargo clean
cargo build --release

DLL size: 2566KB

Older nightly (nightly-2017-07-22):

cargo clean
rustup run nightly-2017-07-22 cargo build --release

DLL size: 1111KB

After stripping them with strip.exe from mingw64, both are around 650KB.

alexcrichton · 2017-11-04T14:54:17Z

@rkarp which project is that for?

rkarp · 2017-11-04T17:49:29Z

I used this repo: https://github.com/rkarp/rust-dll-demo

I turned off LTO for the above test, but that didn't seem to change anything besides all generated DLLs (stripped & non-stripped) being smaller overall. With LTO, the old nightly still produces significantly smaller DLLs. When stripped however, the difference is once again almost gone.

alexcrichton · 2017-11-06T17:29:46Z

@rkarp I think your regression is unrelated, I've opened #45809 to track that.

Looks like this issue is fixed though, so closing.

nagisa added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Oct 29, 2017

Mark-Simulacrum added I-nominated regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. labels Oct 30, 2017

michaelwoerister mentioned this issue Oct 31, 2017

Take crate-type into account when generating symbol export lists (linker version scripts) #45650

Merged

rust-highfive added P-medium Medium priority and removed I-nominated labels Nov 2, 2017

nikomatsakis assigned michaelwoerister Nov 2, 2017

alexcrichton mentioned this issue Nov 6, 2017

cdylib size regression on MinGW #45809

Closed

alexcrichton closed this as completed Nov 6, 2017

nagisa mentioned this issue Nov 21, 2017

Binaries built with 1.21.0 are 5x the size of those from 1.20.0 #46150

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cdylib artifact size boost #45613

cdylib artifact size boost #45613

king6cong commented Oct 29, 2017 •

edited

Loading

nagisa commented Oct 29, 2017

nagisa commented Oct 29, 2017

king6cong commented Oct 29, 2017 •

edited

Loading

Mark-Simulacrum commented Oct 30, 2017

michaelwoerister commented Oct 30, 2017

alexcrichton commented Oct 30, 2017

king6cong commented Oct 31, 2017 •

edited

Loading

alexcrichton commented Oct 31, 2017

alexcrichton commented Oct 31, 2017

king6cong commented Oct 31, 2017 •

edited

Loading

michaelwoerister commented Oct 31, 2017

michaelwoerister commented Nov 2, 2017

nikomatsakis commented Nov 2, 2017

king6cong commented Nov 3, 2017

rkarp commented Nov 3, 2017

alexcrichton commented Nov 4, 2017

rkarp commented Nov 4, 2017

alexcrichton commented Nov 6, 2017

cdylib artifact size boost #45613

cdylib artifact size boost #45613

Comments

king6cong commented Oct 29, 2017 • edited Loading

on nightly-2017-06-30

latest nightly (rustc 1.23.0-nightly (269cf50 2017-10-28))

nagisa commented Oct 29, 2017

nagisa commented Oct 29, 2017

king6cong commented Oct 29, 2017 • edited Loading

on nightly-2017-06-30

latest nightly (rustc 1.23.0-nightly (269cf50 2017-10-28))

Mark-Simulacrum commented Oct 30, 2017

michaelwoerister commented Oct 30, 2017

alexcrichton commented Oct 30, 2017

king6cong commented Oct 31, 2017 • edited Loading

on nightly-2017-06-30

on latest nightly

alexcrichton commented Oct 31, 2017

alexcrichton commented Oct 31, 2017

king6cong commented Oct 31, 2017 • edited Loading

michaelwoerister commented Oct 31, 2017

michaelwoerister commented Nov 2, 2017

nikomatsakis commented Nov 2, 2017

king6cong commented Nov 3, 2017

rkarp commented Nov 3, 2017

alexcrichton commented Nov 4, 2017

rkarp commented Nov 4, 2017

alexcrichton commented Nov 6, 2017

king6cong commented Oct 29, 2017 •

edited

Loading

latest nightly (rustc 1.23.0-nightly (`269cf50` 2017-10-28))

king6cong commented Oct 29, 2017 •

edited

Loading

latest nightly (rustc 1.23.0-nightly (`269cf50` 2017-10-28))

king6cong commented Oct 31, 2017 •

edited

Loading

king6cong commented Oct 31, 2017 •

edited

Loading