document environment variables used when building the compiler

[vitiral]

I am a newbie to compiler development and came across a pretty big documentation gap: environment variables.

I ran the following script to try and pull out all the existing environment variables. I can run it on whatever directories that project contributors want me to. I would like to document these if possible!

Action items for someone more knowledgeable than me:

• I am running against src/lib* src/bootstrap. Are there other directories I should be searching?
• I am searching for env::var(_os) (rust) and os.environ.get (python). Are there other regexes that should be searched?
• Please provide feedback on which of these should be better documented, and if you have a one line "off the top of my head" documentation feel free to write it here.
• Does stage0 complicate things, and we need to be mindful of what environment variables are set when compiling the compiler?
• I think these should probably go in the Configuration section. Let me know if you have a better spot in mind.

rg '^.*?((env::var(_os)?)|(os\.environ\.get))\(([\w"]+)\).*?$' \
    -r '$5' -N --no-filename \
    src/lib* src/bootstrap \
    | sort -u

Output

"APPVEYOR"                                                                                                                                                                                                 
"AR"                    
"BOOTSTRAP_PARENT_ID"
"BOOTSTRAP_PYTHON"     
"BUILD"                                
"CARGO_CFG_TARGET_ARCH"
"CARGO_CFG_TARGET_ENV"
"CARGO_CFG_TARGET_OS"
"CARGO_CFG_TARGET_VENDOR"
"CARGO_TARGET_DIR"     
"CFG_COMPILER_HOST_TRIPLE"
"CFLAGS"            
"DESTDIR"                       
"HOME"                        
"HOST"               
"JEMALLOC_OVERRIDE"
key            
"L4RE_LIBDIR"        
libdir         
"LLVM_CONFIG"  
"LLVM_LINK_SHARED"
"LLVM_RUSTLLVM"        
"LLVM_STATIC_STDCPP"   
"MACOSX_DEPLOYMENT_TARGET"
"MACOSX_STD_DEPLOYMENT_TARGET"
"MAKEFLAGS"     
"MSYSCON"          
"MUSL_ROOT"             
"NUM_JOBS"                 
"OUT_DIR"              
"PAGER"                                
"PATH"                
"RUST_BACKTRACE"   
rustc              
"RUSTC_BOOTSTRAP"
"RUSTC_CODEGEN_UNITS"  
"RUSTC_COLOR"            
"RUSTC_CRT_STATIC"
"RUSTC_DEBUG_ASSERTIONS"        
"RUSTC_DEBUGINFO"
"RUSTC_DEBUGINFO_LINES"
"RUSTC_FORCE_INCR_COMP_ARTIFACT_HEADER"
"RUSTC_FORCE_UNSTABLE"
"RUSTC_HOST_LINKER"  
"RUSTC_INCREMENTAL"
"RUSTC_LIBDIR" 
"RUSTC_METADATA_SUFFIX"
"RUSTC_NO_PREFER_DYNAMIC"
"RUSTC_ON_FAIL"        
"RUSTC_RETRY_LINKER_ON_SEGFAULT"
"RUSTC_RPATH"
"RUSTC_SAVE_ANALYSIS"
"RUSTC_STAGE"
"RUSTC_SYSROOT"
"RUSTC_TARGET_LINKER"
"RUSTC_THINLTO"
"RUSTC_VERBOSE"
"RUST_DEP_GRAPH"
"RUST_DEP_GRAPH_FILTER"
"RUSTDOC_CRATE_VERSION"
"RUSTDOC_REAL"   
"RUST_FORBID_DEP_GRAPH_EDGE"
"RUST_MIN_STACK"
"RUST_REGION_GRAPH"
"RUST_REGION_GRAPH_NODE"
"RUST_SAVE_ANALYSIS_CONFIG"
"RUST_TARGET_PATH"     
"RUST_TEST_NOCAPTURE"                  
"RUST_TEST_STRESS"    
"RUST_TEST_THREADS"
"SCCACHE_ERROR_LOG"
"SCCACHE_PATH"
"SCCACHE_TARGET"       
"SRC"                    
"SUDO_USER"    
"TARGET"                        
"TERM"       
"TERMINFO"           
"TERMINFO_DIRS"
"TEST_DEVICE_ADDR"
"TEST_MIRI"          
"TMPDIR"       
"TRAVIS"       
"USER"          
"USERPROFILE"          
var                    
"WIX"       

[vitiral]

Note: I believe this is dfferent from rust-lang/rust#44074, as this is concerned with the environment variables used for building the compiler itself, not the environment variables that the compiler uses.

However, stage0 probably makes use of the latter. I'm not sure how to manage the fact that those are related -- we probably need to document both.

rust-lang/rust#44074 also brings up the env! macro. Please let me know if there are other ways to get environment variables than env::var(_os) (rust) and os.environ.get (python). Hopefully we don't have too many bash scripts, as it would just look like plain old variables there...

[petrochenkov]

Most of the listed variables are used by the build system and test suite, not compiler itself.
Many of them are also implementation details (they are both set and read internally) that are not supposed to be set from the outside of rustbuild.

(P.S. I suspect the meaning of many of internal variables could could be more clear if naming schemes were more explicit and consistent, e.g. everything used by bootstrap/bin/rustc.rs could be prefixed with RUSTC_WRAPPER_, bootstrap/bin/rustdoc.rs - RUSTDOC_WRAPPER_, etc.)

[theotherjimmy]

the rustc and libdir variables appear to be "RUSTC_STAGE", "RUSTC_REAL", "RUSTC_SNAPSHOT_LIBDIR", "RUSTC_LIBDIR"

    let (rustc, libdir) = if target.is_none() && version.is_none() {
        ("RUSTC_SNAPSHOT", "RUSTC_SNAPSHOT_LIBDIR")
    } else {
        ("RUSTC_REAL", "RUSTC_LIBDIR")
    };
    let stage = env::var("RUSTC_STAGE").expect("RUSTC_STAGE was not set");
    let sysroot = env::var_os("RUSTC_SYSROOT").expect("RUSTC_SYSROOT was not set");
    let mut on_fail = env::var_os("RUSTC_ON_FAIL").map(|of| Command::new(of));

    let rustc = env::var_os(rustc).unwrap_or_else(|| panic!("{:?} was not set", rustc));
    let libdir = env::var_os(libdir).unwrap_or_else(|| panic!("{:?} was not set", libdir));
    let mut dylib_path = bootstrap::util::dylib_path();
    dylib_path.insert(0, PathBuf::from(libdir));

    let mut cmd = Command::new(rustc);
    cmd.args(&args)

[retep998]

Also be wary of code that does use std::env::var; and uses it simply as var.

[steveklabnik]

Triage: we now have a home for this kind of information https://doc.rust-lang.org/rustc/

[pnkfelix]

There was some semi-recent discussion bemoaning the compilers undiscoverable environment variables; I suggested there that it might be a good idea to extend the rustc --help -v output to include a section dedicated to environment variables.

[pnkfelix]

(But in practice if we want such a list to be kept up to date, we'll need an API for defining the environment variables in active use by the compiler, much the same way that we define e.g. command line switches via a standard API...)

[mark-i-m]

Has there been any progress on this?

[mark-i-m]

Fyi

https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/env.20vars.20and.20x.2Epy.20.2338686

[jyn514]

Most of the listed variables are used by the build system and test suite, not compiler itself.
Many of them are also implementation details (they are both set and read internally) that are not supposed to be set from the outside of rustbuild.

I agree with this. Given that we document how to replicate the environment variables for a given run (https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html#environment-variables) I don't think it's important to document the variables themselves - the supported interface here is config.toml, not the env vars. If we were going to document this it would live in src/bootstrap/README.md anyway, not the external facing docs.