Bad codegen partitioning with non-incremental compile in release mode

[jonhoo]

I have a larger application called distributary that encounters a linking error on current nightly when you try to compile mit-pdos/noria@216ec42 (also occurs on earlier commits) in release mode with incremental compilation turned off:

$ rustc --version
rustc 1.28.0-nightly (e3bf634e0 2018-06-28)
$ env CARGO_INCREMENTAL=0 cargo b --release --bin souplet
   Compiling distributary v0.1.0 (file:///home/jon/dev/distributary)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet0-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet1-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet10-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet11-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet12-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet13-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet14-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet15-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet2-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet3-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet4-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet5-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet6-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet7-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet8-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet9-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o" "-o" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b" "/scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.crate.allocator.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/scratch/cargo-target/release/deps" "-L" "/scratch/cargo-target/release/build/backtrace-sys-ed50c3fbaa0d48ec/out" "-L" "/usr/lib" "-L" "/scratch/cargo-target/release/build/librocksdb-sys-84f47854a9cf1b55/out" "-L" "/scratch/cargo-target/release/build/rust-crypto-ced3206b69ce39b0/out" "-L" "/scratch/cargo-target/release/build/miniz-sys-6907519dfa0e351a/out" "-L" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/scratch/cargo-target/release/deps/libdistributary-da37b7f16a76acbb.rlib" "/scratch/cargo-target/release/deps/libstreamunordered-5a9237ca75ad0178.rlib" "/scratch/cargo-target/release/deps/libstream_cancel-9f8b6a0de5b9cccc.rlib" "/scratch/cargo-target/release/deps/libmir-f520571f43a6ec68.rlib" "/scratch/cargo-target/release/deps/libdataflow-201aa72799505388.rlib" "/scratch/cargo-target/release/deps/libtimekeeper-da7c4412fde7cb09.rlib" "/scratch/cargo-target/release/deps/libtempfile-cfb332116a2ff4ea.rlib" "/scratch/cargo-target/release/deps/libremove_dir_all-1972708e2dff8a43.rlib" "/scratch/cargo-target/release/deps/librocksdb-eeb97c0236eaa111.rlib" "/scratch/cargo-target/release/deps/liblibrocksdb_sys-0d6b79a68eb8cc49.rlib" "/scratch/cargo-target/release/deps/libregex-9fd646d3c71ec7eb.rlib" "/scratch/cargo-target/release/deps/libutf8_ranges-c59216e7b48c08cb.rlib" "/scratch/cargo-target/release/deps/libregex_syntax-7aa96219ae904c00.rlib" "/scratch/cargo-target/release/deps/libucd_util-d358e844ee9b3641.rlib" "/scratch/cargo-target/release/deps/libaho_corasick-f261526ac15754b2.rlib" "/scratch/cargo-target/release/deps/libmemchr-9f54cff398456aba.rlib" "/scratch/cargo-target/release/deps/librand-f0be0a99d754832b.rlib" "/scratch/cargo-target/release/deps/librand_core-890633630e1a08df.rlib" "/scratch/cargo-target/release/deps/libitertools-f2822bd97c79bd2b.rlib" "/scratch/cargo-target/release/deps/libeither-e9639929bc9c96da.rlib" "/scratch/cargo-target/release/deps/libevmap-2dcc25e88ce7edf7.rlib" "/scratch/cargo-target/release/deps/librahashmap-1e2082501f4008b1.rlib" "/scratch/cargo-target/release/deps/libapi-19cbdd42be57edbd.rlib" "/scratch/cargo-target/release/deps/libhyper-6cb1e4a581ec1b74.rlib" "/scratch/cargo-target/release/deps/libwant-324dd6dfe1e46291.rlib" "/scratch/cargo-target/release/deps/libtry_lock-b7c5785edc67802c.rlib" "/scratch/cargo-target/release/deps/libhttparse-cc7214eccbdea8a1.rlib" "/scratch/cargo-target/release/deps/libh2-0dcd5b315066cafd.rlib" "/scratch/cargo-target/release/deps/libindexmap-3d2b781d7eaa209d.rlib" "/scratch/cargo-target/release/deps/libstring-208d469b2eb38187.rlib" "/scratch/cargo-target/release/deps/libhttp-82c0a31552817620.rlib" "/scratch/cargo-target/release/deps/libfutures_cpupool-69177f4662e4ff3e.rlib" "/scratch/cargo-target/release/deps/libchannel-0016965ebd38b3e3.rlib" "/scratch/cargo-target/release/deps/libthrottled_reader-7e60943322dad09e.rlib" "/scratch/cargo-target/release/deps/libasync_bincode-df1d28cefed5211d.rlib" "/scratch/cargo-target/release/deps/libtokio-3d33a85e82bb2bc7.rlib" "/scratch/cargo-target/release/deps/libtokio_udp-e580d834189b3963.rlib" "/scratch/cargo-target/release/deps/libtokio_codec-597cc8ed08684a1a.rlib" "/scratch/cargo-target/release/deps/libtokio_tcp-2525b15268d47a37.rlib" "/scratch/cargo-target/release/deps/libtokio_timer-47b7ed1669fe25dc.rlib" "/scratch/cargo-target/release/deps/libtokio_reactor-e670bc42a7ebc209.rlib" "/scratch/cargo-target/release/deps/libtokio_fs-0c19a7ce1bd82b24.rlib" "/scratch/cargo-target/release/deps/libtokio_threadpool-7caded51d25aa4bb.rlib" "/scratch/cargo-target/release/deps/librand-b1353c4410d02bfd.rlib" "/scratch/cargo-target/release/deps/libnum_cpus-67536290ffc5dc69.rlib" "/scratch/cargo-target/release/deps/libcrossbeam_deque-57a511a18d88fa9c.rlib" "/scratch/cargo-target/release/deps/libcrossbeam_epoch-65a2c7adf5cda377.rlib" "/scratch/cargo-target/release/deps/libscopeguard-bae1bab23bc1a9a3.rlib" "/scratch/cargo-target/release/deps/libmemoffset-cd3e4f40a2d099c7.rlib" "/scratch/cargo-target/release/deps/libcrossbeam_utils-efdefb99cf7a5107.rlib" "/scratch/cargo-target/release/deps/libarrayvec-b588f750819e5262.rlib" "/scratch/cargo-target/release/deps/libnodrop-9a0c3995df5e2fbf.rlib" "/scratch/cargo-target/release/deps/libtokio_executor-70618f29d21d845c.rlib" "/scratch/cargo-target/release/deps/libbufstream-38a663efffaac70d.rlib" "/scratch/cargo-target/release/deps/libtokio_io-9b3d821bab7ebc34.rlib" "/scratch/cargo-target/release/deps/libfutures-a5841c464a217433.rlib" "/scratch/cargo-target/release/deps/libbincode-6ca29a2f700bb392.rlib" "/scratch/cargo-target/release/deps/libbasics-d7efdc7acd49e1c1.rlib" "/scratch/cargo-target/release/deps/libpetgraph-0e757162fad3841a.rlib" "/scratch/cargo-target/release/deps/libordermap-53b0295a398c8085.rlib" "/scratch/cargo-target/release/deps/libfixedbitset-5ddef31ba82f5714.rlib" "/scratch/cargo-target/release/deps/libnom_sql-f2edbbd272e020af.rlib" "/scratch/cargo-target/release/deps/libnom-8a0fabe7e7f869b3.rlib" "/scratch/cargo-target/release/deps/libmemchr-ffca46615f647984.rlib" "/scratch/cargo-target/release/deps/libfnv-3f4dd8f6aa5aae23.rlib" "/scratch/cargo-target/release/deps/libarccstr-4573b2a3f5a39fcd.rlib" "/scratch/cargo-target/release/deps/libassert_infrequent-6ba94e943cd0f7fd.rlib" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libtest-044121a3ee9ec9c8.rlib" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libterm-a530b4df1f6b087a.rlib" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgetopts-ebaa9330e19ad094.rlib" "/scratch/cargo-target/release/deps/libconsensus-6eb48c8b2338a842.rlib" "/scratch/cargo-target/release/deps/libzookeeper-c38b7e513051484a.rlib" "/scratch/cargo-target/release/deps/libsnowflake-264b253736130360.rlib" "/scratch/cargo-target/release/deps/libmio_extras-8d68c2bd61408eef.rlib" "/scratch/cargo-target/release/deps/liblazycell-2ad1bf5ff037757e.rlib" "/scratch/cargo-target/release/deps/libmio-7873359cdd37de24.rlib" "/scratch/cargo-target/release/deps/libslab-3c66808c5d553bd5.rlib" "/scratch/cargo-target/release/deps/libnet2-10b9dfdf90bbe580.rlib" "/scratch/cargo-target/release/deps/liblazycell-0402bd591d3290e8.rlib" "/scratch/cargo-target/release/deps/liblog-5ae5c43899104168.rlib" "/scratch/cargo-target/release/deps/liblazy_static-836292728b653c0c.rlib" "/scratch/cargo-target/release/deps/libbytes-2e179691966a1956.rlib" "/scratch/cargo-target/release/deps/libiovec-2664e440b623b43b.rlib" "/scratch/cargo-target/release/deps/libslog_term-12cce1c2a70584b6.rlib" "/scratch/cargo-target/release/deps/libthread_local-a28fd41f31b00339.rlib" "/scratch/cargo-target/release/deps/liblazy_static-6ab320be5c343dd8.rlib" "/scratch/cargo-target/release/deps/libunreachable-9a89aeb7bdf0ff6d.rlib" "/scratch/cargo-target/release/deps/libvoid-c3610640208824a7.rlib" "/scratch/cargo-target/release/deps/libterm-1d86d93672404162.rlib" "/scratch/cargo-target/release/deps/libbyteorder-022f4d9bd8b8257a.rlib" "/scratch/cargo-target/release/deps/libisatty-8b82fa60a41976cf.rlib" "/scratch/cargo-target/release/deps/libchrono-ebe951381ac1187c.rlib" "/scratch/cargo-target/release/deps/libnum_integer-ce927daee7214560.rlib" "/scratch/cargo-target/release/deps/libnum_traits-7d7889bf6603c36d.rlib" "/scratch/cargo-target/release/deps/libtime-6e5c8648e8b69a5c.rlib" "/scratch/cargo-target/release/deps/libslog-55ed2cc25693d855.rlib" "/scratch/cargo-target/release/deps/libserde_json-0ced60ec6e124300.rlib" "/scratch/cargo-target/release/deps/libitoa-e09c11d51d2ced84.rlib" "/scratch/cargo-target/release/deps/libdtoa-ad96437661d0db93.rlib" "/scratch/cargo-target/release/deps/libfailure-4fcc023820c12785.rlib" "/scratch/cargo-target/release/deps/libbacktrace-a0b51d305ff6ae2c.rlib" "/scratch/cargo-target/release/deps/libbacktrace_sys-960f672e6c54a924.rlib" "/scratch/cargo-target/release/deps/librustc_demangle-49c48d56bdbd3c9e.rlib" "/scratch/cargo-target/release/deps/libcfg_if-cb30ad19a978a506.rlib" "/scratch/cargo-target/release/deps/libclap-81b819da4f92f017.rlib" "/scratch/cargo-target/release/deps/libvec_map-dcd18c9a06fe218e.rlib" "/scratch/cargo-target/release/deps/libserde-47931d94c909af06.rlib" "/scratch/cargo-target/release/deps/libtextwrap-9f66ed4e6b0d952a.rlib" "/scratch/cargo-target/release/deps/libunicode_width-a0a53cd11f42fbd9.rlib" "/scratch/cargo-target/release/deps/libstrsim-482b76ff49e8daca.rlib" "/scratch/cargo-target/release/deps/libbitflags-978a75e486f708dd.rlib" "/scratch/cargo-target/release/deps/libatty-c19d004b1ddaf458.rlib" "/scratch/cargo-target/release/deps/liblibc-dca1c89ddaa2bbff.rlib" "/scratch/cargo-target/release/deps/libansi_term-b76564dfb4be04c2.rlib" "-Wl,--start-group" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-f9776412ae7aa499.rlib" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-54f5e90b6163277a.rlib" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-d38cd88231d191e3.rlib" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-d632067cea94a522.rlib" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_system-942a13ea54bd0c51.rlib" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-971850e38acc5f31.rlib" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-e8760697e58c0b14.rlib" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-fb37a4ea1db1e473.rlib" "-Wl,--end-group" "/home/jon/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-3d06b32e587d2669.rlib" "-Wl,-Bdynamic" "-l" "lz4" "-l" "rocksdb" "-l" "stdc++" "-l" "util" "-l" "util" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util" "-l" "util"
  = note: /scratch/cargo-target/release/deps/libdistributary-da37b7f16a76acbb.rlib(distributary-da37b7f16a76acbb.distributary15-801f6c5b9baeaaaf659ccf689438db8f.rs.rcgu.o):(.data.rel.ro._ZN12distributary10controller15listen_internal28_$u7b$$u7b$closure$u7d$$u7d$2RS17h352c4cd07a809e86E+0x0): multiple definition of `distributary::controller::listen_internal::{{closure}}::RS'
          /scratch/cargo-target/release/deps/souplet-d1dd5e17d8eb8c2b.souplet12-c13c77a8e5cfecca9cb4354767d3d260.rs.rcgu.o:(.data.rel.ro._ZN12distributary10controller15listen_internal28_$u7b$$u7b$closure$u7d$$u7d$2RS17h352c4cd07a809e86E+0x0): first defined here
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: Could not compile `distributary`.

Interestingly, it links just fine incremental compilation turned on:

$ env CARGO_INCREMENTAL=1 cargo b --release --bin souplet
   Compiling basics v0.1.0 (file:///home/jon/dev/distributary/basics)
   Compiling consensus v0.1.0 (file:///home/jon/dev/distributary/consensus)
   Compiling channel v0.1.0 (file:///home/jon/dev/distributary/channel)
   Compiling api v0.1.0 (file:///home/jon/dev/distributary/api)
warning: unused import: `assert_infrequent`
 --> api/src/controller.rs:1:5
  |
1 | use assert_infrequent;
  |     ^^^^^^^^^^^^^^^^^
  |
  = note: #[warn(unused_imports)] on by default

   Compiling dataflow v0.1.0 (file:///home/jon/dev/distributary/dataflow)
   Compiling mir v0.1.0 (file:///home/jon/dev/distributary/mir)
   Compiling distributary v0.1.0 (file:///home/jon/dev/distributary)
    Finished release [optimized + debuginfo] target(s) in 1m 14s

It also compiles fine in debug mode with incremental compilation turned off:

$ env CARGO_INCREMENTAL=0 cargo b --bin souplet
   Compiling basics v0.1.0 (file:///home/jon/dev/distributary/basics)
   Compiling consensus v0.1.0 (file:///home/jon/dev/distributary/consensus)
   Compiling channel v0.1.0 (file:///home/jon/dev/distributary/channel)
   Compiling api v0.1.0 (file:///home/jon/dev/distributary/api)
   Compiling dataflow v0.1.0 (file:///home/jon/dev/distributary/dataflow)
   Compiling mir v0.1.0 (file:///home/jon/dev/distributary/mir)
   Compiling distributary v0.1.0 (file:///home/jon/dev/distributary)
    Finished dev [unoptimized + debuginfo] target(s) in 41.34s

The issue also occurs after cargo clean. Unfortunately, due to some NLL issues in past nightlies (#51348 and #51649), bisecting nightlies this isn't trivial. I've also tried producing a minimized reproducing example, but to no avail. This could be a dupe of #47989, although I don't immediately see any duplicated linking targets. Scanning through cargo's output with --verbose, it also looks like the number of linked objects in debug vs release are the same.

https://github.com/mit-pdos/distributary/tree/216ec42058b962727974ac7a0d43c84097f3f73d

[jonhoo]

It also sounds very similar to #36852 (except it occurs with incremental turned off), which suggests it's a bug in the codegen partitioning code.

[jonhoo]

Another data-point:

$ env CARGO_INCREMENTAL=0 RUSTFLAGS='-C codegen-units=1' cargo b --release --bin souplet
...
   Compiling dataflow v0.1.0 (file:///home/jon/dev/distributary/dataflow)
   Compiling mir v0.1.0 (file:///home/jon/dev/distributary/mir)
   Compiling distributary v0.1.0 (file:///home/jon/dev/distributary)
    Finished release [optimized + debuginfo] target(s) in 10m 36s

So this is indeed related to multiple codegen units.

[weihanglo]

FWIW, the bug still exists as of Rust 1.84.0, but with the dev profile.