Memory usage regression in #49045

[ishitatsuyuki]

On the CircleCI Docker remote machine which has 2 cores and 8G memory, I'm no longer able to compile rustc. This error consistently occurs in stage1. This was bisected to #49045.

It's likely that overcommit is disabled and vfork() resulted in rejection, although I wasn't able to obtain any further traces about this.

The possible cause may be:

• Simply that the memory requirement increased.
• We're leaking memory / freeing too late. rustc shouldn't need much memory at the stage it would invoke linker.

cc @Zoxc

Compiling syntax_ext v0.0.0 (file:///home/user/rust/src/libsyntax_ext)
error: could not exec the linker `cc`
  |
  = note: Cannot allocate memory (os error 12)
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc-d162f1a13c8ef190.rustc0-20c527915fa1ef6087fc31cbe68d7f2f.rs.rcgu.o" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc-d162f1a13c8ef190.rustc1-20c527915fa1ef6087fc31cbe68d7f2f.rs.rcgu.o" "-o" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc-d162f1a13c8ef190.so" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc-d162f1a13c8ef190.crate.metadata.rcgu.o" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc-d162f1a13c8ef190.crate.allocator.rcgu.o" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/backtrace-sys-340c7157c7761734/out/.libs" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/miniz-sys-bf90fe466cd4a9fa/out" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "test-902559d4a0cc83d5" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "term-36880752958d2c6f" "-Wl,-Bstatic" "-Wl,--whole-archive" "/tmp/rustc.GuL2KuWri9zo/libflate2-75c9cc82bb755b48.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.GuL2KuWri9zo/libminiz_sys-3d17c9d844239fa3.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.GuL2KuWri9zo/libbacktrace-786cb2b6a87aed68.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.GuL2KuWri9zo/libbacktrace_sys-60501d5f3e4fb8f6.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.GuL2KuWri9zo/librustc_demangle-ffb675b42151a6f8.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.GuL2KuWri9zo/libbyteorder-f652c706c62d3d96.rlib" "-Wl,--no-whole-archive" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-Wl,-Bdynamic" "-l" "proc_macro-50a63500c1ea1d55" "-Wl,-Bstatic" "-Wl,--whole-archive" "/tmp/rustc.GuL2KuWri9zo/libjobserver-5189829dca651532.rlib" "-Wl,--no-whole-archive" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-Wl,-Bdynamic" "-l" "rustc_const_math-b2c60d57ea59ca96" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "rustc_back-a514fb34c8a54105" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "syntax-915450af937848b5" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "rustc_errors-d6070e053e9037b0" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "syntax_pos-e425942a0298756c" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "rustc_data_structures-9ce9b8b9893796c6" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "serialize-b63c42e100577192" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "rustc_cratesio_shim-45d3d77a26fe81d5" "-Wl,-Bstatic" "-Wl,--whole-archive" "/tmp/rustc.GuL2KuWri9zo/liblazy_static-d0629a75f2783580.rlib" "-Wl,--no-whole-archive" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-Wl,-Bdynamic" "-l" "graphviz-06bb7008d45cdff0" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "fmt_macros-7fa0522df5af949a" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps" "-l" "arena-657cf6916cd0a947" "-Wl,--start-group" "-L" "/home/user/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "std-0feb32955e89b888" "-Wl,--end-group" "-Wl,-Bstatic" "/tmp/rustc.GuL2KuWri9zo/libcompiler_builtins-825ad861341732a3.rlib" "-Wl,-Bdynamic" "-l" "util" "-l" "util" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util" "-l" "util" "-shared" "-Wl,-rpath,$ORIGIN/../lib"

error: aborting due to previous error

error: Could not compile `rustc`.

[Zoxc]

It doesn't look like it #49045 creates crazy memory regressions at least: http://perf.rust-lang.org/compare.html?start=56714acc5eb0687ed9a7566fdebe5528657fc5b3&end=7222241e7c2d7caf9ad6ee6e34748e4addfb8dd3&stat=max-rss

Could you run this with -Z time-passes?
Adding the following to src/bootstrap/bin/rustc.rs should to the trick:

        if stage == "1" && crate_name == "rustc" {
            cmd.arg("-Z").arg("time-passes");
            let file = std::fs::File::create("rustc-timings").unwrap();
            cmd.stdout(std::process::Stdio::from(file));
        }

[BusyJay]

After upgrading to rustc 1.27.0-nightly (48fa6f963 2018-04-05), pingcap/tikv is hard to be built in release mode on my dev machine. It starts four rustc processes, each of them occupies more than 2G memory. Our docker service used to build tikv fails too when two jobs are run concurrently which are run on a machine with 16G memory.

The version we used before upgrading was nightly-2018-01-12, which worked fine.

[ishitatsuyuki]

@BusyJay Interesting. What error message do you encounter? Is it same as what described in this issue?

Also, can you try 2018-04-04 which should be before the problematic change?

[BusyJay]

What I saw was:

Out of memory: Kill process 12260 (rustc) score 212 or sacrifice child
Killed process 12260 (rustc) total-vm:4881248kB, anon-rss:3458380kB, file-rss:0kB, shmem-rss:0kB

error: Could not compile `tikv`.
warning: build failed, waiting for other jobs to finish...

OOM exists since rustc 1.25.0-nightly (bacb5c58d 2018-01-26). May not be the same issue, I can create a new issue for this.

[ishitatsuyuki]

@BusyJay I think that's a different issue. Please create an new issue optionally with bisecting the bad build.

[nikomatsakis]

It seems like we need more data before we can figure out how to proceed -- -Ztime-passes might be a start, it'd also be nice if we had a smaller test case that does succeed but also shows a higher peak memory, so we can profile the memory consumption or something.

[nikomatsakis]

I guess that this is a problem bootstrapping, so we ought to be able to measure the memory usage requirement locally.

[nikomatsakis]

@Zoxc are you able to do a little probing?

[nikomatsakis]

triage: P-high

Classifying as P-high to try and get more information on what is happening. Assigning Zoxc for the time being.

[Zoxc]

#50102 landed which should fix the memory usage regression in #49045. The memory usage for queries shouldn't matter that much as we free that memory before invoking the linker.

[ishitatsuyuki]

Confirmed that this is fixed by #50102. Thanks @Zoxc!