"Standard library (libstd, liballoc, libcore, libtest) guide"

[kennytm]

Some main points raised by rust-lang/rust#50466:

• How to build and test the library without needing to rebuild the compiler every time.

  • developing from stage0: one could use ./x.py test --stage 0 --no-doc src/libstd to quickly build and test the standard library, if features from a pre-built compiler is sufficient.
  • developing from stage1 (e.g. a new intrinsic is added): Suggest the proper --keep-stage syntax so the compiler will only be built once and then kept frozen?

• Code arrangement

  • For libcore the tests must be doc tests or integrated tests in src/libcore/tests/

[gnzlbg]

You forgot liballoc, but yeah, I think this book should have a section about it somewhere. I just landed here to report that https://rust-lang-nursery.github.io/rustc-guide/tests/running.html does not show how to run liballoc tests

[mark-i-m]

@kennytm @gnzlbg would rather if either of you be able to make a PR? It would be much appreciated!

EDIT: autocorrect stinks :(

[gnzlbg]

I know nothing and must be doing everything wrong:

• it takes forever to build liballoc for me (~1 h)
• it takes forever to edit-compile-debug liballoc for me
• running before/after benchmarks takes forever because I need to build it twice, and it takes 2x forever (~2 h)
• I can't have two rustc folders because each folder takes ~20-30 Gb of HDD space, etc

So I am more in the avoid touching any std library component as much as possible camp, and also, in the camp of people that would benefit most from all this information.

[gnzlbg]

For example, I just wanted to run the vec benchmarks of the liballoc crate, and ./x.py bench -h doesn't return anything meaningful. So I don't even know how to run only the vec benchmarks there.

[mark-i-m]

Hmm... that's interesting. I do a lot of no_std stuff, and I don't recall liballoc or libcore taking more than a couple of minutes to compile (i5-7500T, 8GB RAM, SSD drive). I'm totally guessing, but

• Are you using a network filesystem? That is really slow in my experience.
• What compiler debugging options turned on in your config toml? I've seen some of these blow up compile time. And if you are only working on libraries, maybe that's a waste.

[gnzlbg]

Are you using a network filesystem? That is really slow in my experience

I have an SSD, I am running, for example, ./x.py bench src/liballoc, and that compiles the whole rustc first in release mode (which duplicates my hdd space usage) and then runs all benchmarks, instead of only the ones I am interested about.

What compiler debugging options turned on in your config toml? I've seen some of these blow up compile time. And if you are only working on libraries, maybe that's a waste.

I work on both, but I am willing to have two different config.toml files for when working only on the library. This is my config.toml: https://gist.github.com/gnzlbg/a3fe5be837986773b3b7204726eb83b3

[mark-i-m]

./x.py bench src/liballoc, and that compiles the whole rustc first in release mode (which duplicates my hdd space usage) and then runs all benchmarks, instead of only the ones I am interested about.

I think this kind of makes sense. Since the compiler also uses these libraries, if you change one, you are changing a dependency of the compiler, which could in theory change the performance of the compiler or the code it produces.

Perhaps using --stage 1 might be good enough for quick prototyping? @kennytm Is that the purpose of the flags you mentioned in the OP?

This is my config.toml

IIRC, when I turned on rust.debug-assertions and llvm.assertions things got slow for me too (though, this was a while back...). Perhaps try turning them off?

[mark-i-m]

Does this section cover any of the things you wanted: https://rust-lang-nursery.github.io/rustc-guide/how-to-build-and-run.html#suggested-workflows-for-faster-builds-of-the-compiler

[gnzlbg]

That does look really nice. Thanks!

[mark-i-m]

Is there anything you think should be added to that section?

[gnzlbg]

The only thing that pops to mind is that I don't know how to use git bisect to do what's mentioned there, but I am unsure of whether that's worth including. When I'll need that I'll just google for it :)

[mark-i-m]

cc #78 "Bisecting against PRs"

[dwijnand]

I know nothing and must be doing everything wrong:

• it takes forever to build liballoc for me (~1 h)
• it takes forever to edit-compile-debug liballoc for me
• running before/after benchmarks takes forever because I need to build it twice, and it takes 2x forever (~2 h)
• I can't have two rustc folders because each folder takes ~20-30 Gb of HDD space, etc

So I am more in the avoid touching any std library component as much as possible camp, and also, in the camp of people that would benefit most from all this information.

That's me, too. I recently learnt that part of the problem was a misunderstanding of the stages, so I was testing at stage 1 instead of stage 0: rust-lang/rust#57963

With regards to disk space, I wonder if it's possible reuse the LLVM part, for both disk usage and build time reasons: #276

[mark-i-m]

That's me, too. I recently learnt that part of the problem was a misunderstanding of the stages, so I was testing at stage 1 instead of stage 0

As I mentioned on the other thread, a PR would be much appreciated.

cc the related #201

[aloucks]

The #1 item is incredibly useful. It's a shame that it's buried in this issue and not up-to-date on:
https://rust-lang.github.io/rustc-guide/tests/running.html

@kennytm Do you have an example for #2?

1. developing from stage0: one could use ./x.py test --stage 0 --no-doc src/libstd to quickly build and test the standard library, if features from a pre-built compiler is sufficient.
2. developing from stage1 (e.g. a new intrinsic is added): Suggest the proper --keep-stage syntax so the compiler will only be built once and then kept frozen?

[golddranks]

Hi, is there any up-to-date info how to run quickly tests for stdlib or liballoc without bootstrapping the compiler? I'm not touching the compiler itself, I'd like just to have a quick sanity checks while developing stdlib APIs. I tried: ./x.py test --stage 0 --no-doc src/liballoc, but it errors out with a huge number of `error[E0063]: missing field test_type in initializer of test::TestDesc errors. I'm using a recent nightly (2019-10-03).

[mark-i-m]

@golddranks does this help? https://rust-lang.github.io/rustc-guide/how-to-build-and-run.html#suggested-workflows-for-faster-builds-of-the-compiler

Otherwise, perhaps @Mark-Simulacrum has suggestions?

[Mark-Simulacrum]

That's your best bet command these days, but as you've noted it doesn't quite work if the bootstrap compiler you're using (typically beta) has diverged its test executable sufficiently from what we're using in-tree. When you say "I'm using a recent nightly (2019-10-03)." I'm not sure what you mean, x.py typically doesn't look in your environment for a rustc at all.

You might be able to resolve this behavior by setting rustc to something more recent (e.g., $HOME/.rustup/toolchains/nightly.*/bin/rustc, though expanded), but it's basically not guaranteed to work out of the box.

[golddranks]

Ah, sorry, I meant that my default compiler (via rustup) is a recent nightly, but it seems that x.py manages it's own toolchain, so never mind. Thanks for the pointers, I'll try them out.

[rylev]

Triage: closing. Now that https://github.com/rust-lang/std-dev-guide exists, we will no longer have the need to document standard library development in this guide.