Link unstable features

[albertlarsan68]

Helps with #103548

[rustbot]

r? @jyn514

(rustbot has picked a reviewer for you, use r? to override)

[albertlarsan68]

@jyn514 This PR is ready.
The only problem that is left, is that the docs of the crates core, alloc, std, proc_macro, test are re-built every time x test tidy is run.

It is just the docs for those crates, not their dependencies, but I think the caching/freshness check is broken some[how,where]?

[bors]

☔ The latest upstream changes (presumably #106757) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #107143) made this pull request unmergeable. Please resolve the merge conflicts.

[albertlarsan68]

@jyn514 I rebased and added the missing data to make this change pass.
@rustbot ready

[jyn514]

Can you see how long it takes to run tidy before and after this change? Please include the time it takes to run rustdoc -wjson on the standard library.

[jyn514]

Sorry for the long delay btw; if you want to reassign this to Mark I'm ok with that.

[jyn514]

this looks broadly right to me except for the stage issue :)

[jyn514]

Oh right, I'd forgotten that you have a slow laptop 😓

Hmm, given that it's such a large slowdown, maybe we should move the check out of tidy? We can put it in the test::UnstableBook step (you'll have to pull it out of the test_book macro, but that seems ok).

[albertlarsan68]

@jyn514 I have moved the generation of the pages to unstable-book-gen, so now the src/doc/unstable-book/src/library-features only contains the optional notes about the feature. So now, there is no tidy changes, and the style is consistent.
I switched to using the in-tree rustdoc-json-types, because they are being generated with stage 2 in CI. This means that a simple x doc unstable-book will be likely to fail, but I think not having to have logic to handle two versions and bump the rustdoc-types dependency at each bootstrap compile bump is a plus. I had this logic at one point, but I removed it to support only the in-tree rustdoc-json-types. Do you want me to add it back?
I will try to make a diff of the markdown output of unstable-book-gen with and without this change, but the results are likely to be fuzzy / irrelevant.
@rustbot ready

[jyn514]

I don't think I'll have time to review this in the near future, unfortunately.

r? bootstrap

[albertlarsan68]

No problem, do what you think is the best.

Welcome @ozkanonur to this PR!

[onur-ozkan]

Greatk work! 🔥

I have some suggestions for this implementation:

[onur-ozkan]

This can be defined outside of the loop.

Also, we can reduce the nested scopes with the following way

// Given a `NestedMeta` like `feature = "xyz"`, returns `xyz`.
let get_feature_name = |nested: &_| match nested {
    NestedMeta::Meta(Meta::NameValue(name_value)) => {
        if !is_ident(name_value.path.get_ident()?, "feature") {
            return None;
        }
        match &name_value.lit {
            Lit::Str(s) => Some(s.value()),
            _ => None,
        }
    }
    _ => None,
};

[onur-ozkan]

Is it only me that this looks hard to debug/understand the workflow inside? :)

Maybe we can have some comments to inform developer about what's going on in there. + Using expect instead of unwrap can be nice to have as well.

[albertlarsan68]

This tries (quite badly) to generate the URL that the item belongs to.
If there is a way to get it directly from Rustdoc, I will happily use it.
Concerning the unwraps, this is mostly prototyping code that is not meant to last.

[onur-ozkan]

This tries (quite badly) to generate the URL that the item belongs to.
If there is a way to get it directly from Rustdoc, I will happily use it.

Having comments which explains aim of the function instructions(which are complicated to read/track) can be handy imo.(e.g

rust/src/bootstrap/builder.rs

Lines 1165 to 1930 in 91eb6f9

	/// Prepares an invocation of `cargo` to be run.
	///
	/// This will create a `Command` that represents a pending execution of
	/// Cargo. This cargo will be configured to use `compiler` as the actual
	/// rustc compiler, its output will be scoped by `mode`'s output directory,
	/// it will pass the `--target` flag for the specified `target`, and will be
	/// executing the Cargo command `cmd`.
	pub fn cargo(
	&self,
	compiler: Compiler,
	mode: Mode,
	source_type: SourceType,
	target: TargetSelection,
	cmd: &str,
	) -> Cargo {
	let mut cargo = self.bare_cargo(compiler, mode, target, cmd);
	let out_dir = self.stage_out(compiler, mode);
	// Codegen backends are not yet tracked by -Zbinary-dep-depinfo,
	// so we need to explicitly clear out if they've been updated.
	for backend in self.codegen_backends(compiler) {
	self.clear_if_dirty(&out_dir, &backend);
	}
	if cmd == "doc" || cmd == "rustdoc" {
	let my_out = match mode {
	// This is the intended out directory for compiler documentation.
	Mode::Rustc | Mode::ToolRustc => self.compiler_doc_out(target),
	Mode::Std => {
	if self.config.cmd.json() {
	out_dir.join(target.triple).join("json-doc")
	} else {
	out_dir.join(target.triple).join("doc")
	}
	}
	_ => panic!("doc mode {:?} not expected", mode),
	};
	let rustdoc = self.rustdoc(compiler);
	self.clear_if_dirty(&my_out, &rustdoc);
	}
	let profile_var = |name: &str| {
	let profile = if self.config.rust_optimize { "RELEASE" } else { "DEV" };
	format!("CARGO_PROFILE_{}_{}", profile, name)
	};
	// See comment in rustc_llvm/build.rs for why this is necessary, largely llvm-config
	// needs to not accidentally link to libLLVM in stage0/lib.
	cargo.env("REAL_LIBRARY_PATH_VAR", &util::dylib_path_var());
	if let Some(e) = env::var_os(util::dylib_path_var()) {
	cargo.env("REAL_LIBRARY_PATH", e);
	}
	// Set a flag for `check`/`clippy`/`fix`, so that certain build
	// scripts can do less work (i.e. not building/requiring LLVM).
	if cmd == "check" || cmd == "clippy" || cmd == "fix" {
	// If we've not yet built LLVM, or it's stale, then bust
	// the rustc_llvm cache. That will always work, even though it
	// may mean that on the next non-check build we'll need to rebuild
	// rustc_llvm. But if LLVM is stale, that'll be a tiny amount
	// of work comparatively, and we'd likely need to rebuild it anyway,
	// so that's okay.
	if crate::native::prebuilt_llvm_config(self, target).is_err() {
	cargo.env("RUST_CHECK", "1");
	}
	}
	let stage = if compiler.stage == 0 && self.local_rebuild {
	// Assume the local-rebuild rustc already has stage1 features.
	1
	} else {
	compiler.stage
	};
	let mut rustflags = Rustflags::new(target);
	if stage != 0 {
	if let Ok(s) = env::var("CARGOFLAGS_NOT_BOOTSTRAP") {
	cargo.args(s.split_whitespace());
	}
	rustflags.env("RUSTFLAGS_NOT_BOOTSTRAP");
	} else {
	if let Ok(s) = env::var("CARGOFLAGS_BOOTSTRAP") {
	cargo.args(s.split_whitespace());
	}
	rustflags.env("RUSTFLAGS_BOOTSTRAP");
	if cmd == "clippy" {
	// clippy overwrites sysroot if we pass it to cargo.
	// Pass it directly to clippy instead.
	// NOTE: this can't be fixed in clippy because we explicitly don't set `RUSTC`,
	// so it has no way of knowing the sysroot.
	rustflags.arg("--sysroot");
	rustflags.arg(
	self.sysroot(compiler)
	.as_os_str()
	.to_str()
	.expect("sysroot must be valid UTF-8"),
	);
	// Only run clippy on a very limited subset of crates (in particular, not build scripts).
	cargo.arg("-Zunstable-options");
	// Explicitly does *not* set `--cfg=bootstrap`, since we're using a nightly clippy.
	let host_version = Command::new("rustc").arg("--version").output().map_err(|_| ());
	let output = host_version.and_then(|output| {
	if output.status.success() {
	Ok(output)
	} else {
	Err(())
	}
	}).unwrap_or_else(|_| {
	eprintln!(
	"error: `x.py clippy` requires a host `rustc` toolchain with the `clippy` component"
	);
	eprintln!("help: try `rustup component add clippy`");
	crate::detail_exit(1);
	});
	if !t!(std::str::from_utf8(&output.stdout)).contains("nightly") {
	rustflags.arg("--cfg=bootstrap");
	}
	} else {
	rustflags.arg("--cfg=bootstrap");
	}
	}
	let use_new_symbol_mangling = match self.config.rust_new_symbol_mangling {
	Some(setting) => {
	// If an explicit setting is given, use that
	setting
	}
	None => {
	if mode == Mode::Std {
	// The standard library defaults to the legacy scheme
	false
	} else {
	// The compiler and tools default to the new scheme
	true
	}
	}
	};
	if use_new_symbol_mangling {
	rustflags.arg("-Csymbol-mangling-version=v0");
	} else {
	rustflags.arg("-Csymbol-mangling-version=legacy");
	rustflags.arg("-Zunstable-options");
	}
	// Enable cfg checking of cargo features for everything but std and also enable cfg
	// checking of names and values.
	//
	// Note: `std`, `alloc` and `core` imports some dependencies by #[path] (like
	// backtrace, core_simd, std_float, ...), those dependencies have their own
	// features but cargo isn't involved in the #[path] process and so cannot pass the
	// complete list of features, so for that reason we don't enable checking of
	// features for std crates.
	cargo.arg(if mode != Mode::Std {
	"-Zcheck-cfg=names,values,output,features"
	} else {
	"-Zcheck-cfg=names,values,output"
	});
	// Add extra cfg not defined in/by rustc
	//
	// Note: Altrough it would seems that "-Zunstable-options" to `rustflags` is useless as
	// cargo would implicitly add it, it was discover that sometimes bootstrap only use
	// `rustflags` without `cargo` making it required.
	rustflags.arg("-Zunstable-options");
	for (restricted_mode, name, values) in EXTRA_CHECK_CFGS {
	if *restricted_mode == None || *restricted_mode == Some(mode) {
	// Creating a string of the values by concatenating each value:
	// ',"tvos","watchos"' or '' (nothing) when there are no values
	let values = match values {
	Some(values) => values
	.iter()
	.map(|val| [",", "\"", val, "\""])
	.flatten()
	.collect::<String>(),
	None => String::new(),
	};
	rustflags.arg(&format!("--check-cfg=values({name}{values})"));
	}
	}
	// FIXME: It might be better to use the same value for both `RUSTFLAGS` and `RUSTDOCFLAGS`,
	// but this breaks CI. At the very least, stage0 `rustdoc` needs `--cfg bootstrap`. See
	// #71458.
	let mut rustdocflags = rustflags.clone();
	rustdocflags.propagate_cargo_env("RUSTDOCFLAGS");
	if stage == 0 {
	rustdocflags.env("RUSTDOCFLAGS_BOOTSTRAP");
	} else {
	rustdocflags.env("RUSTDOCFLAGS_NOT_BOOTSTRAP");
	}
	if let Ok(s) = env::var("CARGOFLAGS") {
	cargo.args(s.split_whitespace());
	}
	match mode {
	Mode::Std | Mode::ToolBootstrap | Mode::ToolStd => {}
	Mode::Rustc | Mode::Codegen | Mode::ToolRustc => {
	// Build proc macros both for the host and the target
	if target != compiler.host && cmd != "check" {
	cargo.arg("-Zdual-proc-macros");
	rustflags.arg("-Zdual-proc-macros");
	}
	}
	}
	// This tells Cargo (and in turn, rustc) to output more complete
	// dependency information. Most importantly for rustbuild, this
	// includes sysroot artifacts, like libstd, which means that we don't
	// need to track those in rustbuild (an error prone process!). This
	// feature is currently unstable as there may be some bugs and such, but
	// it represents a big improvement in rustbuild's reliability on
	// rebuilds, so we're using it here.
	//
	// For some additional context, see #63470 (the PR originally adding
	// this), as well as #63012 which is the tracking issue for this
	// feature on the rustc side.
	cargo.arg("-Zbinary-dep-depinfo");
	let allow_features = match mode {
	Mode::ToolBootstrap | Mode::ToolStd => {
	// Restrict the allowed features so we don't depend on nightly
	// accidentally.
	//
	// binary-dep-depinfo is used by rustbuild itself for all
	// compilations.
	//
	// Lots of tools depend on proc_macro2 and proc-macro-error.
	// Those have build scripts which assume nightly features are
	// available if the `rustc` version is "nighty" or "dev". See
	// bin/rustc.rs for why that is a problem. Instead of labeling
	// those features for each individual tool that needs them,
	// just blanket allow them here.
	//
	// If this is ever removed, be sure to add something else in
	// its place to keep the restrictions in place (or make a way
	// to unset RUSTC_BOOTSTRAP).
	"binary-dep-depinfo,proc_macro_span,proc_macro_span_shrink,proc_macro_diagnostic"
	.to_string()
	}
	Mode::Std | Mode::Rustc | Mode::Codegen | Mode::ToolRustc => String::new(),
	};
	cargo.arg("-j").arg(self.jobs().to_string());
	// FIXME: Temporary fix for https://github.com/rust-lang/cargo/issues/3005
	// Force cargo to output binaries with disambiguating hashes in the name
	let mut metadata = if compiler.stage == 0 {
	// Treat stage0 like a special channel, whether it's a normal prior-
	// release rustc or a local rebuild with the same version, so we
	// never mix these libraries by accident.
	"bootstrap".to_string()
	} else {
	self.config.channel.to_string()
	};
	// We want to make sure that none of the dependencies between
	// std/test/rustc unify with one another. This is done for weird linkage
	// reasons but the gist of the problem is that if librustc, libtest, and
	// libstd all depend on libc from crates.io (which they actually do) we
	// want to make sure they all get distinct versions. Things get really
	// weird if we try to unify all these dependencies right now, namely
	// around how many times the library is linked in dynamic libraries and
	// such. If rustc were a static executable or if we didn't ship dylibs
	// this wouldn't be a problem, but we do, so it is. This is in general
	// just here to make sure things build right. If you can remove this and
	// things still build right, please do!
	match mode {
	Mode::Std => metadata.push_str("std"),
	// When we're building rustc tools, they're built with a search path
	// that contains things built during the rustc build. For example,
	// bitflags is built during the rustc build, and is a dependency of
	// rustdoc as well. We're building rustdoc in a different target
	// directory, though, which means that Cargo will rebuild the
	// dependency. When we go on to build rustdoc, we'll look for
	// bitflags, and find two different copies: one built during the
	// rustc step and one that we just built. This isn't always a
	// problem, somehow -- not really clear why -- but we know that this
	// fixes things.
	Mode::ToolRustc => metadata.push_str("tool-rustc"),
	// Same for codegen backends.
	Mode::Codegen => metadata.push_str("codegen"),
	_ => {}
	}
	cargo.env("__CARGO_DEFAULT_LIB_METADATA", &metadata);
	if cmd == "clippy" {
	rustflags.arg("-Zforce-unstable-if-unmarked");
	}
	rustflags.arg("-Zmacro-backtrace");
	let want_rustdoc = self.doc_tests != DocTests::No;
	// We synthetically interpret a stage0 compiler used to build tools as a
	// "raw" compiler in that it's the exact snapshot we download. Normally
	// the stage0 build means it uses libraries build by the stage0
	// compiler, but for tools we just use the precompiled libraries that
	// we've downloaded
	let use_snapshot = mode == Mode::ToolBootstrap;
	assert!(!use_snapshot || stage == 0 || self.local_rebuild);
	let maybe_sysroot = self.sysroot(compiler);
	let sysroot = if use_snapshot { self.rustc_snapshot_sysroot() } else { &maybe_sysroot };
	let libdir = self.rustc_libdir(compiler);
	// Clear the output directory if the real rustc we're using has changed;
	// Cargo cannot detect this as it thinks rustc is bootstrap/debug/rustc.
	//
	// Avoid doing this during dry run as that usually means the relevant
	// compiler is not yet linked/copied properly.
	//
	// Only clear out the directory if we're compiling std; otherwise, we
	// should let Cargo take care of things for us (via depdep info)
	if !self.config.dry_run() && mode == Mode::Std && cmd == "build" {
	self.clear_if_dirty(&out_dir, &self.rustc(compiler));
	}
	// Customize the compiler we're running. Specify the compiler to cargo
	// as our shim and then pass it some various options used to configure
	// how the actual compiler itself is called.
	//
	// These variables are primarily all read by
	// src/bootstrap/bin/{rustc.rs,rustdoc.rs}
	cargo
	.env("RUSTBUILD_NATIVE_DIR", self.native_dir(target))
	.env("RUSTC_REAL", self.rustc(compiler))
	.env("RUSTC_STAGE", stage.to_string())
	.env("RUSTC_SYSROOT", &sysroot)
	.env("RUSTC_LIBDIR", &libdir)
	.env("RUSTDOC", self.bootstrap_out.join("rustdoc"))
	.env(
	"RUSTDOC_REAL",
	if cmd == "doc" || cmd == "rustdoc" || (cmd == "test" && want_rustdoc) {
	self.rustdoc(compiler)
	} else {
	PathBuf::from("/path/to/nowhere/rustdoc/not/required")
	},
	)
	.env("RUSTC_ERROR_METADATA_DST", self.extended_error_dir())
	.env("RUSTC_BREAK_ON_ICE", "1");
	// Clippy support is a hack and uses the default `cargo-clippy` in path.
	// Don't override RUSTC so that the `cargo-clippy` in path will be run.
	if cmd != "clippy" {
	cargo.env("RUSTC", self.bootstrap_out.join("rustc"));
	}
	// Dealing with rpath here is a little special, so let's go into some
	// detail. First off, `-rpath` is a linker option on Unix platforms
	// which adds to the runtime dynamic loader path when looking for
	// dynamic libraries. We use this by default on Unix platforms to ensure
	// that our nightlies behave the same on Windows, that is they work out
	// of the box. This can be disabled, of course, but basically that's why
	// we're gated on RUSTC_RPATH here.
	//
	// Ok, so the astute might be wondering "why isn't `-C rpath` used
	// here?" and that is indeed a good question to ask. This codegen
	// option is the compiler's current interface to generating an rpath.
	// Unfortunately it doesn't quite suffice for us. The flag currently
	// takes no value as an argument, so the compiler calculates what it
	// should pass to the linker as `-rpath`. This unfortunately is based on
	// the **compile time** directory structure which when building with
	// Cargo will be very different than the runtime directory structure.
	//
	// All that's a really long winded way of saying that if we use
	// `-Crpath` then the executables generated have the wrong rpath of
	// something like `$ORIGIN/deps` when in fact the way we distribute
	// rustc requires the rpath to be `$ORIGIN/../lib`.
	//
	// So, all in all, to set up the correct rpath we pass the linker
	// argument manually via `-C link-args=-Wl,-rpath,...`. Plus isn't it
	// fun to pass a flag to a tool to pass a flag to pass a flag to a tool
	// to change a flag in a binary?
	if self.config.rust_rpath && util::use_host_linker(target) {
	let rpath = if target.contains("apple") {
	// Note that we need to take one extra step on macOS to also pass
	// `-Wl,-instal_name,@rpath/...` to get things to work right. To
	// do that we pass a weird flag to the compiler to get it to do
	// so. Note that this is definitely a hack, and we should likely
	// flesh out rpath support more fully in the future.
	rustflags.arg("-Zosx-rpath-install-name");
	Some("-Wl,-rpath,@loader_path/../lib")
	} else if !target.contains("windows") {
	rustflags.arg("-Clink-args=-Wl,-z,origin");
	Some("-Wl,-rpath,$ORIGIN/../lib")
	} else {
	None
	};
	if let Some(rpath) = rpath {
	rustflags.arg(&format!("-Clink-args={}", rpath));
	}
	}
	if let Some(host_linker) = self.linker(compiler.host) {
	cargo.env("RUSTC_HOST_LINKER", host_linker);
	}
	if self.is_fuse_ld_lld(compiler.host) {
	cargo.env("RUSTC_HOST_FUSE_LD_LLD", "1");
	cargo.env("RUSTDOC_FUSE_LD_LLD", "1");
	}
	if let Some(target_linker) = self.linker(target) {
	let target = crate::envify(&target.triple);
	cargo.env(&format!("CARGO_TARGET_{}_LINKER", target), target_linker);
	}
	if self.is_fuse_ld_lld(target) {
	rustflags.arg("-Clink-args=-fuse-ld=lld");
	}
	self.lld_flags(target).for_each(|flag| {
	rustdocflags.arg(&flag);
	});
	if !(["build", "check", "clippy", "fix", "rustc"].contains(&cmd)) && want_rustdoc {
	cargo.env("RUSTDOC_LIBDIR", self.rustc_libdir(compiler));
	}
	let debuginfo_level = match mode {
	Mode::Rustc | Mode::Codegen => self.config.rust_debuginfo_level_rustc,
	Mode::Std => self.config.rust_debuginfo_level_std,
	Mode::ToolBootstrap | Mode::ToolStd | Mode::ToolRustc => {
	self.config.rust_debuginfo_level_tools
	}
	};
	cargo.env(profile_var("DEBUG"), debuginfo_level.to_string());
	cargo.env(
	profile_var("DEBUG_ASSERTIONS"),
	if mode == Mode::Std {
	self.config.rust_debug_assertions_std.to_string()
	} else {
	self.config.rust_debug_assertions.to_string()
	},
	);
	cargo.env(
	profile_var("OVERFLOW_CHECKS"),
	if mode == Mode::Std {
	self.config.rust_overflow_checks_std.to_string()
	} else {
	self.config.rust_overflow_checks.to_string()
	},
	);
	let split_debuginfo_is_stable = target.contains("linux")
	|| target.contains("apple")
	|| (target.contains("msvc")
	&& self.config.rust_split_debuginfo == SplitDebuginfo::Packed)
	|| (target.contains("windows")
	&& self.config.rust_split_debuginfo == SplitDebuginfo::Off);
	if !split_debuginfo_is_stable {
	rustflags.arg("-Zunstable-options");
	}
	match self.config.rust_split_debuginfo {
	SplitDebuginfo::Packed => rustflags.arg("-Csplit-debuginfo=packed"),
	SplitDebuginfo::Unpacked => rustflags.arg("-Csplit-debuginfo=unpacked"),
	SplitDebuginfo::Off => rustflags.arg("-Csplit-debuginfo=off"),
	};
	if self.config.cmd.bless() {
	// Bless `expect!` tests.
	cargo.env("UPDATE_EXPECT", "1");
	}
	if !mode.is_tool() {
	cargo.env("RUSTC_FORCE_UNSTABLE", "1");
	}
	if let Some(x) = self.crt_static(target) {
	if x {
	rustflags.arg("-Ctarget-feature=+crt-static");
	} else {
	rustflags.arg("-Ctarget-feature=-crt-static");
	}
	}
	if let Some(x) = self.crt_static(compiler.host) {
	cargo.env("RUSTC_HOST_CRT_STATIC", x.to_string());
	}
	if let Some(map_to) = self.build.debuginfo_map_to(GitRepo::Rustc) {
	let map = format!("{}={}", self.build.src.display(), map_to);
	cargo.env("RUSTC_DEBUGINFO_MAP", map);
	// `rustc` needs to know the virtual `/rustc/$hash` we're mapping to,
	// in order to opportunistically reverse it later.
	cargo.env("CFG_VIRTUAL_RUST_SOURCE_BASE_DIR", map_to);
	}
	// Enable usage of unstable features
	cargo.env("RUSTC_BOOTSTRAP", "1");
	self.add_rust_test_threads(&mut cargo);
	// Almost all of the crates that we compile as part of the bootstrap may
	// have a build script, including the standard library. To compile a
	// build script, however, it itself needs a standard library! This
	// introduces a bit of a pickle when we're compiling the standard
	// library itself.
	//
	// To work around this we actually end up using the snapshot compiler
	// (stage0) for compiling build scripts of the standard library itself.
	// The stage0 compiler is guaranteed to have a libstd available for use.
	//
	// For other crates, however, we know that we've already got a standard
	// library up and running, so we can use the normal compiler to compile
	// build scripts in that situation.
	if mode == Mode::Std {
	cargo
	.env("RUSTC_SNAPSHOT", &self.initial_rustc)
	.env("RUSTC_SNAPSHOT_LIBDIR", self.rustc_snapshot_libdir());
	} else {
	cargo
	.env("RUSTC_SNAPSHOT", self.rustc(compiler))
	.env("RUSTC_SNAPSHOT_LIBDIR", self.rustc_libdir(compiler));
	}
	// Tools that use compiler libraries may inherit the `-lLLVM` link
	// requirement, but the `-L` library path is not propagated across
	// separate Cargo projects. We can add LLVM's library path to the
	// platform-specific environment variable as a workaround.
	if mode == Mode::ToolRustc || mode == Mode::Codegen {
	if let Some(llvm_config) = self.llvm_config(target) {
	let llvm_libdir = output(Command::new(&llvm_config).arg("--libdir"));
	add_link_lib_path(vec![llvm_libdir.trim().into()], &mut cargo);
	}
	}
	// Compile everything except libraries and proc macros with the more
	// efficient initial-exec TLS model. This doesn't work with `dlopen`,
	// so we can't use it by default in general, but we can use it for tools
	// and our own internal libraries.
	if !mode.must_support_dlopen() && !target.triple.starts_with("powerpc-") {
	cargo.env("RUSTC_TLS_MODEL_INITIAL_EXEC", "1");
	}
	if self.config.incremental {
	cargo.env("CARGO_INCREMENTAL", "1");
	} else {
	// Don't rely on any default setting for incr. comp. in Cargo
	cargo.env("CARGO_INCREMENTAL", "0");
	}
	if let Some(ref on_fail) = self.config.on_fail {
	cargo.env("RUSTC_ON_FAIL", on_fail);
	}
	if self.config.print_step_timings {
	cargo.env("RUSTC_PRINT_STEP_TIMINGS", "1");
	}
	if self.config.print_step_rusage {
	cargo.env("RUSTC_PRINT_STEP_RUSAGE", "1");
	}
	if self.config.backtrace_on_ice {
	cargo.env("RUSTC_BACKTRACE_ON_ICE", "1");
	}
	cargo.env("RUSTC_VERBOSE", self.verbosity.to_string());
	if source_type == SourceType::InTree {
	let mut lint_flags = Vec::new();
	// When extending this list, add the new lints to the RUSTFLAGS of the
	// build_bootstrap function of src/bootstrap/bootstrap.py as well as
	// some code doesn't go through this `rustc` wrapper.
	lint_flags.push("-Wrust_2018_idioms");
	lint_flags.push("-Wunused_lifetimes");
	lint_flags.push("-Wsemicolon_in_expressions_from_macros");
	if self.config.deny_warnings {
	lint_flags.push("-Dwarnings");
	rustdocflags.arg("-Dwarnings");
	}
	// This does not use RUSTFLAGS due to caching issues with Cargo.
	// Clippy is treated as an "in tree" tool, but shares the same
	// cache as other "submodule" tools. With these options set in
	// RUSTFLAGS, that causes *every* shared dependency to be rebuilt.
	// By injecting this into the rustc wrapper, this circumvents
	// Cargo's fingerprint detection. This is fine because lint flags
	// are always ignored in dependencies. Eventually this should be
	// fixed via better support from Cargo.
	cargo.env("RUSTC_LINT_FLAGS", lint_flags.join(" "));
	rustdocflags.arg("-Wrustdoc::invalid_codeblock_attributes");
	}
	if mode == Mode::Rustc {
	rustflags.arg("-Zunstable-options");
	rustflags.arg("-Wrustc::internal");
	}
	// Throughout the build Cargo can execute a number of build scripts
	// compiling C/C++ code and we need to pass compilers, archivers, flags, etc
	// obtained previously to those build scripts.
	// Build scripts use either the `cc` crate or `configure/make` so we pass
	// the options through environment variables that are fetched and understood by both.
	//
	// FIXME: the guard against msvc shouldn't need to be here
	if target.contains("msvc") {
	if let Some(ref cl) = self.config.llvm_clang_cl {
	cargo.env("CC", cl).env("CXX", cl);
	}
	} else {
	let ccache = self.config.ccache.as_ref();
	let ccacheify = |s: &Path| {
	let ccache = match ccache {
	Some(ref s) => s,
	None => return s.display().to_string(),
	};
	// FIXME: the cc-rs crate only recognizes the literal strings
	// `ccache` and `sccache` when doing caching compilations, so we
	// mirror that here. It should probably be fixed upstream to
	// accept a new env var or otherwise work with custom ccache
	// vars.
	match &ccache[..] {
	"ccache" | "sccache" => format!("{} {}", ccache, s.display()),
	_ => s.display().to_string(),
	}
	};
	let triple_underscored = target.triple.replace("-", "_");
	let cc = ccacheify(&self.cc(target));
	cargo.env(format!("CC_{}", triple_underscored), &cc);
	let cflags = self.cflags(target, GitRepo::Rustc, CLang::C).join(" ");
	cargo.env(format!("CFLAGS_{}", triple_underscored), &cflags);
	if let Some(ar) = self.ar(target) {
	let ranlib = format!("{} s", ar.display());
	cargo
	.env(format!("AR_{}", triple_underscored), ar)
	.env(format!("RANLIB_{}", triple_underscored), ranlib);
	}
	if let Ok(cxx) = self.cxx(target) {
	let cxx = ccacheify(&cxx);
	let cxxflags = self.cflags(target, GitRepo::Rustc, CLang::Cxx).join(" ");
	cargo
	.env(format!("CXX_{}", triple_underscored), &cxx)
	.env(format!("CXXFLAGS_{}", triple_underscored), cxxflags);
	}
	}
	if mode == Mode::Std && self.config.extended && compiler.is_final_stage(self) {
	rustflags.arg("-Zsave-analysis");
	cargo.env(
	"RUST_SAVE_ANALYSIS_CONFIG",
	"{\"output_file\": null,\"full_docs\": false,\
	\"pub_only\": true,\"reachable_only\": false,\
	\"distro_crate\": true,\"signatures\": false,\"borrow_data\": false}",
	);
	}
	// If Control Flow Guard is enabled, pass the `control-flow-guard` flag to rustc
	// when compiling the standard library, since this might be linked into the final outputs
	// produced by rustc. Since this mitigation is only available on Windows, only enable it
	// for the standard library in case the compiler is run on a non-Windows platform.
	// This is not needed for stage 0 artifacts because these will only be used for building
	// the stage 1 compiler.
	if cfg!(windows)
	&& mode == Mode::Std
	&& self.config.control_flow_guard
	&& compiler.stage >= 1
	{
	rustflags.arg("-Ccontrol-flow-guard");
	}
	// For `cargo doc` invocations, make rustdoc print the Rust version into the docs
	// This replaces spaces with newlines because RUSTDOCFLAGS does not
	// support arguments with regular spaces. Hopefully someday Cargo will
	// have space support.
	let rust_version = self.rust_version().replace(' ', "\n");
	rustdocflags.arg("--crate-version").arg(&rust_version);
	// Environment variables *required* throughout the build
	//
	// FIXME: should update code to not require this env var
	cargo.env("CFG_COMPILER_HOST_TRIPLE", target.triple);
	// Set this for all builds to make sure doc builds also get it.
	cargo.env("CFG_RELEASE_CHANNEL", &self.config.channel);
	// This one's a bit tricky. As of the time of this writing the compiler
	// links to the `winapi` crate on crates.io. This crate provides raw
	// bindings to Windows system functions, sort of like libc does for
	// Unix. This crate also, however, provides "import libraries" for the
	// MinGW targets. There's an import library per dll in the windows
	// distribution which is what's linked to. These custom import libraries
	// are used because the winapi crate can reference Windows functions not
	// present in the MinGW import libraries.
	//
	// For example MinGW may ship libdbghelp.a, but it may not have
	// references to all the functions in the dbghelp dll. Instead the
	// custom import library for dbghelp in the winapi crates has all this
	// information.
	//
	// Unfortunately for us though the import libraries are linked by
	// default via `-ldylib=winapi_foo`. That is, they're linked with the
	// `dylib` type with a `winapi_` prefix (so the winapi ones don't
	// conflict with the system MinGW ones). This consequently means that
	// the binaries we ship of things like rustc_codegen_llvm (aka the rustc_codegen_llvm
	// DLL) when linked against *again*, for example with procedural macros
	// or plugins, will trigger the propagation logic of `-ldylib`, passing
	// `-lwinapi_foo` to the linker again. This isn't actually available in
	// our distribution, however, so the link fails.
	//
	// To solve this problem we tell winapi to not use its bundled import
	// libraries. This means that it will link to the system MinGW import
	// libraries by default, and the `-ldylib=foo` directives will still get
	// passed to the final linker, but they'll look like `-lfoo` which can
	// be resolved because MinGW has the import library. The downside is we
	// don't get newer functions from Windows, but we don't use any of them
	// anyway.
	if !mode.is_tool() {
	cargo.env("WINAPI_NO_BUNDLED_LIBRARIES", "1");
	}
	for _ in 0..self.verbosity {
	cargo.arg("-v");
	}
	match (mode, self.config.rust_codegen_units_std, self.config.rust_codegen_units) {
	(Mode::Std, Some(n), _) | (_, _, Some(n)) => {
	cargo.env(profile_var("CODEGEN_UNITS"), n.to_string());
	}
	_ => {
	// Don't set anything
	}
	}
	if self.config.locked_deps {
	cargo.arg("--locked");
	}
	if self.config.vendor || self.is_sudo {
	cargo.arg("--frozen");
	}
	// Try to use a sysroot-relative bindir, in case it was configured absolutely.
	cargo.env("RUSTC_INSTALL_BINDIR", self.config.bindir_relative());
	self.ci_env.force_coloring_in_ci(&mut cargo);
	// When we build Rust dylibs they're all intended for intermediate
	// usage, so make sure we pass the -Cprefer-dynamic flag instead of
	// linking all deps statically into the dylib.
	if matches!(mode, Mode::Std | Mode::Rustc) {
	rustflags.arg("-Cprefer-dynamic");
	}
	// When building incrementally we default to a lower ThinLTO import limit
	// (unless explicitly specified otherwise). This will produce a somewhat
	// slower code but give way better compile times.
	{
	let limit = match self.config.rust_thin_lto_import_instr_limit {
	Some(limit) => Some(limit),
	None if self.config.incremental => Some(10),
	_ => None,
	};
	if let Some(limit) = limit {
	if stage == 0 || self.config.default_codegen_backend().unwrap_or_default() == "llvm"
	{
	rustflags.arg(&format!("-Cllvm-args=-import-instr-limit={}", limit));
	}
	}
	}
	Cargo { command: cargo, rustflags, rustdocflags, allow_features }
	}

)

Concerning the unwraps, this is mostly prototyping code that is not meant to last.

It's just to give more clear information in context. Not so necessary I guess.

[bors]

☔ The latest upstream changes (presumably #113508) made this pull request unmergeable. Please resolve the merge conflicts.

[onur-ozkan]

@albertlarsan68 do we have any plans for this PR?

[albertlarsan68]

The thing that I am having a hard time with is generating the links. The machinery to detect what is gated is in place, but the links are hard to generate.

[Dylan-DPC]

@albertlarsan68 any updates on this?

[albertlarsan68]

Nothing new.
This PR has the code needed to generate at least the (human-readable) paths to unstable items, but I wasn't able to generate correct links to them.
Maybe I'll remove the link generation code, then we could merge this as a "helps with" instead of a "fixes".

[rust-log-analyzer]

The job x86_64-gnu-llvm-15 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

GITHUB_ACTION=__run_7
GITHUB_ACTIONS=true
GITHUB_ACTION_REF=
GITHUB_ACTION_REPOSITORY=
GITHUB_ACTOR=albertlarsan68
GITHUB_API_URL=https://api.github.com
GITHUB_BASE_REF=master
GITHUB_ENV=/home/runner/work/_temp/_runner_file_commands/set_env_d24457cd-05d4-475d-9d33-23aff4415249
GITHUB_EVENT_NAME=pull_request
---
GITHUB_SERVER_URL=https://github.com
GITHUB_SHA=b6dcf11b49d74b22d09282ca4d9892752a7561dc
GITHUB_STATE=/home/runner/work/_temp/_runner_file_commands/save_state_d24457cd-05d4-475d-9d33-23aff4415249
GITHUB_STEP_SUMMARY=/home/runner/work/_temp/_runner_file_commands/step_summary_d24457cd-05d4-475d-9d33-23aff4415249
GITHUB_TRIGGERING_ACTOR=albertlarsan68
GITHUB_WORKFLOW_REF=rust-lang/rust/.github/workflows/ci.yml@refs/pull/106681/merge
GITHUB_WORKFLOW_SHA=b6dcf11b49d74b22d09282ca4d9892752a7561dc
GITHUB_WORKSPACE=/home/runner/work/rust/rust
GOROOT_1_19_X64=/opt/hostedtoolcache/go/1.19.13/x64
---
   Compiling unstable-book-gen v0.1.0 (/checkout/src/tools/unstable-book-gen)
error[E0599]: no variant or associated item named `Typedef` found for enum `ItemKind` in the current scope
  --> src/tools/unstable-book-gen/src/main.rs:45:39
   |
45 |         rustdoc_json_types::ItemKind::Typedef => "type",
   |                                       ^^^^^^^ variant or associated item not found in `ItemKind`
For more information about this error, try `rustc --explain E0599`.
For more information about this error, try `rustc --explain E0599`.
error: could not compile `unstable-book-gen` (bin "unstable-book-gen") due to previous error
  local time: Tue Oct 17 17:33:29 UTC 2023
  network time: Tue, 17 Oct 2023 17:33:29 GMT
##[error]Process completed with exit code 1.
Post job cleanup.

[bors]

☔ The latest upstream changes (presumably #120491) made this pull request unmergeable. Please resolve the merge conflicts.

[Dylan-DPC]

@albertlarsan68 if you can rebase this and resolve the conflicts , we can push this forward for a review

[JohnCSimon]

@albertlarsan68
Ping from triage: I'm closing this due to inactivity, Please reopen when you are ready to continue with this.
Note: if you are going to continue please open the PR BEFORE you push to it, else you won't be able to reopen - this is a quirk of github.
Thanks for your contribution.

@rustbot label: +S-inactive