rustc fails to raise rlimits for stacksize on mac for doctest runs through make

[mitsuhiko]

I get rlimit errors when running cargo test on a mac through make:

ERROR 2018-07-28T07:03:41Z: rustc_driver: in_rustc_thread: error calling setrlimit: Invalid argument (os error 22)

If run without make it works. Seems to be this call that is failing:

rust/src/librustc_driver/lib.rs

Lines 1523 to 1528 in 4f1e235

	if libc::setrlimit(libc::RLIMIT_STACK, &mut rlim) != 0 {
	let err = io::Error::last_os_error();
	error!("in_rustc_thread: error calling setrlimit: {}", err);
	std::rt::update_stack_guard();
	true
	} else {

$ ulimit -s
8192

Repro case is just an empty library with a single doctest:

/// ```rust
/// assert!(1 == 1);
/// ```
pub fn foo() {
}

And then this makefile:

test:
	cargo test

Test runs:

mitsuhiko at argus in /tmp/x on git:master? rust 1.27.2
$ cargo test
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running target/debug/deps/x-b63d1d867374a2db

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests x

running 1 test
test src/lib.rs - foo (line 1) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out


mitsuhiko at argus in /tmp/x on git:master? rust 1.27.2
$ make test
cargo test
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running target/debug/deps/x-b63d1d867374a2db

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests x

running 1 test
ERROR 2018-07-28T07:10:59Z: rustc_driver: in_rustc_thread: error calling setrlimit: Invalid argument (os error 22)
test src/lib.rs - foo (line 1) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Versions:

$ (cargo --version; rustc --version; uname -a)
cargo 1.27.0 (1e95190e5 2018-05-27)
rustc 1.27.2 (58cc626de 2018-07-18)
Darwin argus.local 17.7.0 Darwin Kernel Version 17.7.0: Fri Jul  6 19:54:51 PDT 2018; root:xnu-4570.71.3~2/RELEASE_X86_64 x86_64

[mitsuhiko]

It seems that make has a higher ulimit for the stack for some reason:

mitsuhiko at argus in /tmp
$ cat foo.make
print-limits:
	ulimit -s

mitsuhiko at argus in /tmp
$ ulimit -s; make -f foo.make
8192
ulimit -s
65532

[nagisa]

Seems to be another regression similar to #52577.

[nikomatsakis]

Is the failure perhaps because make has already raised the ulimit higher, so our attempts to "raise it" are actually trying to decrease it? Should we just check the current value before invoking setrlimit?

[nikomatsakis]

Compiler-team meeting: Marking as P-medium for now.