Docker Nightly Build Breaks on 1.40 but fine on 1.39

[PrismaPhonic]

Hi,

We are currently using nightly rust for a product demo at my company, and 1.40 breaks our musl build:

   Compiling doggo-api v0.1.0 (/doggo-api)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-nostdlib" "crt1.o" "crti.o" "-L" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.0.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.1.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.10.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.11.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.12.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.13.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.14.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.15.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.2.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.3.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.4.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.5.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.6.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.7.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.8.rcgu.o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.doggo_api.e22cp5fr-cgu.9.rcgu.o" "-o" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c" "/target/x86_64-unknown-linux-musl/release/deps/doggo_api-1c643eb23367362c.wx10qmj6z5yzx4g.rcgu.o" "-Wl,--gc-sections" "-no-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/target/x86_64-unknown-linux-musl/release/deps" "-L" "/target/release/deps" "-L" "/target/x86_64-unknown-linux-musl/release/build/backtrace-sys-40290d965f5ba339/out" "-L" "/target/x86_64-unknown-linux-musl/release/build/ring-ecc4a95f265173e0/out" "-L" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/target/x86_64-unknown-linux-musl/release/deps/libdoggo_api-b6fee484211e04a8.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libdoggo_infra-4484d9ee7f3a183d.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libmysql-0da45d513bf476e4.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libnix-e7a02ee3b293a9be.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libvoid-6664ed4f2ec87196.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libnet2-f3d4518377514f7a.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libflate2-f867a49907693b8b.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libminiz_oxide-370964db15804103.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libadler32-aa50c51e0d58ec27.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libcrc32fast-61e809998ec8dcf8.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libbufstream-690494148f8dfce0.rlib" "/target/x86_64-unknown-linux-musl/release/deps/liburl-bf483bd366bf76b5.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libidna-50262ac246dc99a2.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libpercent_encoding-c0d2d31039372e81.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libfnv-6a9540fc2b94ffc9.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libmysql_common-769f22ee59a8b030.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libbit_vec-6ede940272db9218.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libsha2-75010981d31770fb.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libdigest-8c07184f7b62665a.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libopaque_debug-19f5f8f4c2027ed4.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libfake_simd-2d9cbc108bb34f45.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libblock_buffer-d52fe7e2ae214f2d.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libgeneric_array-b7e844f19f6dd6da.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libtypenum-f036088d18542352.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libblock_padding-fb0807f4829f35d5.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libbyte_tools-2ef921e2181abc50.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libtwox_hash-e2208e5022fb5bfa.rlib" "/target/x86_64-unknown-linux-musl/release/deps/liblexical-35548f2d6c7d8007.rlib" "/target/x86_64-unknown-linux-musl/release/deps/liblexical_core-85e088a6f58382e4.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libarrayvec-64aa4b0b71d42443.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libnodrop-fcc454ec082732f3.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libstatic_assertions-db4d3075529bc603.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libsha1-e72b1fb092daaec7.rlib" "/target/x86_64-unknown-linux-musl/release/deps/librand-1014295af2467a7a.rlib" "/target/x86_64-unknown-linux-musl/release/deps/librand_chacha-8ee0764a689c4e5e.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libc2_chacha-59b8e02e3e70ec89.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libppv_lite86-87d2026d3ab5572f.rlib" "/target/x86_64-unknown-linux-musl/release/deps/librand_core-2bfdfa52e038e94d.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libgetrandom-f2eac7f8665e7d4a.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libuuid-faf5b5cbea0737cd.rlib" "/target/x86_64-unknown-linux-musl/release/deps/librust_decimal-0d1e8f865b9bd691.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libnum-4c98a39ed2561dc2.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libnum_rational-3c4cb9bfdb74e0a6.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libnum_iter-3eab13daddc79240.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libnum_complex-d7013b7eb915a666.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libchrono-03be80e51d366bf8.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libbigdecimal-708444061694fe4d.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libnum_bigint-e2828db91705fc8d.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libnum_integer-2a2b6f00536186bb.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libnum_traits-2b32befad2e6fefe.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libbitflags-a8c3c0865dc41f35.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libsnafu-e9938c57961249f9.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libdoc_comment-471e42119d737fe3.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libbacktrace-b7b71e34f02c40b6.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libbacktrace_sys-9c40379d8d231f8a.rlib" "/target/x86_64-unknown-linux-musl/release/deps/librustc_demangle-6247da6b117824bb.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libdoggo_core-a6c899f46fa15e54.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libdomain_patterns-e9384bde037263cb.rlib" "/target/x86_64-unknown-linux-musl/release/deps/librocket_contrib-c7cbe9c86c1bab67.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libhandlebars-a4832e76653d62a0.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libwalkdir-59c2d9983c784fa3.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libsame_file-5d2815bd7fd4cd16.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libquick_error-0df2ac6cfe705905.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libpest-d79eb69c9e31040b.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libucd_trie-c07447bd7b876b22.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libglob-c4af62d87dc3f458.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libserde_json-fa338db52794c9f0.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libryu-b9b193b944f19c8c.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libitoa-247341888f8e1f54.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libdotenv-f59eeccd9d0eb706.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libfailure-64ecd6727ae1f68c.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libregex-b40cabbe6c91fbac.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libthread_local-dc1258258e974fc5.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libregex_syntax-ee3f41aa4e2c4473.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libaho_corasick-9dd1e4e53ab43dd3.rlib" "/target/x86_64-unknown-linux-musl/release/deps/librocket-75a74aa1b78d4580.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libatty-6a3b8feec050352f.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libbase64-2890995528116058.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libmemchr-e9e8ed9ad274d07b.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libtoml-b3d673d4e8a8009b.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libserde-675a93c40b72e5fc.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libyansi-edf98c88d6b5f9a5.rlib" "/target/x86_64-unknown-linux-musl/release/deps/librocket_http-2ba6d4d0ddc5743c.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libhyper-57d961da04bfaa75.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libmime-3182c6953016b9d8.rlib" "/target/x86_64-unknown-linux-musl/release/deps/liblog-1704a9755545bfa2.rlib" "/target/x86_64-unknown-linux-musl/release/deps/liblog-efbd8a8ab462c778.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libcfg_if-151eba0a73a342b5.rlib" "/target/x86_64-unknown-linux-musl/release/deps/liblanguage_tags-4b99cb7e943ab923.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libtypeable-92c1aa8f2f0279f6.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libtraitobject-cb133943880996d3.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libnum_cpus-e521fb26d5ac6bd8.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libhttparse-8fa27ef85c0ed23f.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libunicase-e4c00389fc8a88f7.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libunicode_xid-69ed1cb55cd370c5.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libstate-546e4c86f106ef30.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libindexmap-0053baa0b53bb1ee.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libcookie-c9cead6e208670e5.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libbase64-eafd3d7c0e1bdb9c.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libsafemem-36f55d95ed6d87ee.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libbyteorder-2dc3d5b12177770e.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libring-c73f1e38881d2e37.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libuntrusted-ad133ecfeaa6c5e8.rlib" "/target/x86_64-unknown-linux-musl/release/deps/liblazy_static-f131710e230a5798.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libtime-07688876f97c9092.rlib" "/target/x86_64-unknown-linux-musl/release/deps/liblibc-6a0b083a57d876d5.rlib" "/target/x86_64-unknown-linux-musl/release/deps/liburl-da61c0423038db9a.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libidna-8e25cbfd56470a11.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libunicode_normalization-dce675b4e522919c.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libunicode_bidi-5bb396dad716e415.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libmatches-41a9c3a78de37b1f.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libsmallvec-86eea87684371830.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libpercent_encoding-72b93100cc0c5006.rlib" "/target/x86_64-unknown-linux-musl/release/deps/libpear-1ea90ccd9dd1b7b8.rlib" "-Wl,--start-group" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-45e1236322efc18f.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-4d66690e2cac514f.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-268591cfdf3e3b49.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-620245b7a07e1660.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace-3e46ac2392c8bcdd.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace_sys-d4dc13409bef6e66.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-dd038e67457bac88.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-a7d701c611a3c472.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-6af7a023fe78d8ea.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-4d94e7b2b49dd594.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-6690b341d1dfc2ef.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-18ea3b1a2c3112ba.rlib" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-b4b650e87163c71f.rlib" "-Wl,--end-group" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-fa571615550699ff.rlib" "-static" "-Wl,-Bdynamic" "crtn.o"
  = note: cc: error: crt1.o: No such file or directory
          cc: error: crti.o: No such file or directory
          cc: error: crtn.o: No such file or directory
          

error: aborting due to previous error

error: could not compile `doggo-api`.

To learn more, run the command again with --verbose.
The command '/bin/sh -c cargo build --target x86_64-unknown-linux-musl --release' returned a non-zero code: 101
make: *** [Makefile:6: build] Error 101

Forcing docker to use 1.39 nightly resolves the issue entirely. Not quite sure what's causing it to break.

Here's the project: https://github.com/planetscale/goodest-doggo

If you clone it and change the Dockerfile to not install and override the specific nightly version you can reproduce the breakage. After modifying the dockerfile just run make build from the root dir.

[mati865]

Delicate of #65335, should be fixed on latest nightly.

[PrismaPhonic]

I made this issue the moment this happened so unless there a newer nightly between the time of me posting and you commenting? Regardless it's typically good GitHub ettiquite to not close issues until you've confirmed with the poster that the issue is resolved on their end. Will check today

[mati865]

I verified it doesn't happen any more about 2 hours later. Since the issues were closed I haven't replied.

I made this issue the moment this happened so unless there a newer nightly between the time of me posting and you commenting?

I don't exact time but nightly was released but AFAIK release process starts on 00:00 UTC and takes at least 2 hours, so it's rather close to the time you posted.
Another thing to take in account is Docker cache. If step where nightly is downloaded hasn't changed Docker could have taken older nightly from the cache.

[PrismaPhonic]

Still breaking on my end with nightly. Please re-open this issue @pietroalbini. I'm not sure what rust-lang policy is, but I would suggest not prematurely closing issues in the future.

[mati865]

@PrismaPhonic as I wrote above it's possibly Docker cache picking up old nightly.
What is output of rustc -vV in your Docker script?

[PrismaPhonic]

Ahh, that was it. Cleared out my docker cache and rebuilt and we are good to go now. Thanks for the help!