diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index f5e1163545d..c9eb512103f 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -19,7 +19,7 @@ jobs: - run: rustup component add rustfmt - run: cargo fmt --all -- --check - run: | - for manifest in `find crates -name Cargo.toml` + for manifest in `find crates benches/benchsuite benches/capture -name Cargo.toml` do echo check fmt for $manifest cargo fmt --all --manifest-path $manifest -- --check @@ -79,6 +79,15 @@ jobs: if: matrix.os == 'macos-latest' - run: cargo build --manifest-path crates/credential/cargo-credential-wincred/Cargo.toml if: matrix.os == 'windows-latest' + - name: Check benchmarks + env: + # Share the target dir to try to cache a few build-time deps. + CARGO_TARGET_DIR: target + run: | + # This only tests one benchmark since it can take over 10 minutes to + # download all workspaces. + cargo test --manifest-path benches/benchsuite/Cargo.toml --all-targets -- cargo + cargo check --manifest-path benches/capture/Cargo.toml - name: Fetch smoke test run: ci/fetch-smoke-test.sh diff --git a/benches/README.md b/benches/README.md new file mode 100644 index 00000000000..b4b8b190a0a --- /dev/null +++ b/benches/README.md @@ -0,0 +1,124 @@ +# Cargo Benchmarking + +This directory contains some benchmarks for cargo itself. This uses +[Criterion] for running benchmarks. It is recommended to read the Criterion +book to get familiar with how to use it. A basic usage would be: + +```sh +cd benches/benchsuite +cargo bench +``` + +The tests involve downloading the index and benchmarking against some +real-world and artificial workspaces located in the [`workspaces`](workspaces) +directory. + +**Beware** that the initial download can take a fairly long amount of time (10 +minutes minimum on an extremely fast network) and require significant disk +space (around 4.5GB). The benchsuite will cache the index and downloaded +crates in the `target/tmp/bench` directory, so subsequent runs should be +faster. You can (and probably should) specify individual benchmarks to run to +narrow it down to a more reasonable set, for example: + +```sh +cargo bench -- resolve_ws/rust +``` + +This will only download what's necessary for the rust-lang/rust workspace +(which is about 330MB) and run the benchmarks against it (which should take +about a minute). To get a list of all the benchmarks, run: + +```sh +cargo bench -- --list +``` + +## Viewing reports + +The benchmarks display some basic information on the command-line while they +run. A more complete HTML report can be found at +`target/criterion/report/index.html` which contains links to all the +benchmarks and summaries. Check out the Criterion book for more information on +the extensive reporting capabilities. + +## Comparing implementations + +Knowing the raw numbers can be useful, but what you're probably most +interested in is checking if your changes help or hurt performance. To do +that, you need to run the benchmarks multiple times. + +First, run the benchmarks from the master branch of cargo without any changes. +To make it easier to compare, Criterion supports naming the baseline so that +you can iterate on your code and compare against it multiple times. + +```sh +cargo bench -- --save-baseline master +``` + +Now you can switch to your branch with your changes. Re-run the benchmarks +compared against the baseline: + +```sh +cargo bench -- --baseline master +``` + +You can repeat the last command as you make changes to re-compare against the +master baseline. + +Without the baseline arguments, it will compare against the last run, which +can be helpful for comparing incremental changes. + +## Capturing workspaces + +The [`workspaces`](workspaces) directory contains several workspaces that +provide a variety of different workspaces intended to provide good exercises +for benchmarks. Some of these are shadow copies of real-world workspaces. This +is done with the tool in the [`capture`](capture) directory. The tool will +copy `Cargo.lock` and all of the `Cargo.toml` files of the workspace members. +It also adds an empty `lib.rs` so Cargo won't error, and sanitizes the +`Cargo.toml` to some degree, removing unwanted elements. Finally, it +compresses everything into a `tgz`. + +To run it, do: + +```sh +cd benches/capture +cargo run -- /path/to/workspace/foo +``` + +The resolver benchmarks also support the `CARGO_BENCH_WORKSPACES` environment +variable, which you can point to a Cargo workspace if you want to try +different workspaces. For example: + +```sh +CARGO_BENCH_WORKSPACES=/path/to/some/workspace cargo bench +``` + +## TODO + +This is just a start for establishing a benchmarking suite for Cargo. There's +a lot that can be added. Some ideas: + +* Fix the benchmarks so that the resolver setup doesn't run every iteration. +* Benchmark [this section of + code](https://github.com/rust-lang/cargo/blob/a821e2cb24d7b6013433f069ab3bad53d160e100/src/cargo/ops/cargo_compile.rs#L470-L549) + which builds the unit graph. The performance there isn't great, and it would + be good to keep an eye on it. Unfortunately that would mean doing a bit of + work to make `generate_targets` publicly visible, and there is a bunch of + setup code that may need to be duplicated. +* Benchmark the fingerprinting code. +* Benchmark running the `cargo` executable. Running something like `cargo + build` or `cargo check` with everything "Fresh" would be a good end-to-end + exercise to measure the overall overhead of Cargo. +* Benchmark pathological resolver scenarios. There might be some cases where + the resolver can spend a significant amount of time. It would be good to + identify if these exist, and create benchmarks for them. This may require + creating an artificial index, similar to the `resolver-tests`. This should + also consider scenarios where the resolver ultimately fails. +* Benchmark without `Cargo.lock`. I'm not sure if this is particularly + valuable, since we are mostly concerned with incremental builds which will + always have a lock file. +* Benchmark just + [`resolve::resolve`](https://github.com/rust-lang/cargo/blob/a821e2cb24d7b6013433f069ab3bad53d160e100/src/cargo/core/resolver/mod.rs#L122) + without anything else. This can help focus on just the resolver. + +[Criterion]: https://bheisler.github.io/criterion.rs/book/ diff --git a/benches/benchsuite/Cargo.toml b/benches/benchsuite/Cargo.toml new file mode 100644 index 00000000000..5c0995d27aa --- /dev/null +++ b/benches/benchsuite/Cargo.toml @@ -0,0 +1,21 @@ +[package] +name = "benchsuite" +version = "0.1.0" +edition = "2018" +license = "MIT OR Apache-2.0" +homepage = "https://github.com/rust-lang/cargo" +repository = "https://github.com/rust-lang/cargo" +documentation = "https://docs.rs/cargo-platform" +description = "Benchmarking suite for Cargo." + +[dependencies] +cargo = { path = "../.." } +# Consider removing html_reports in 0.4 and switching to `cargo criterion`. +criterion = { version = "0.3.5", features = ["html_reports"] } +flate2 = { version = "1.0.3", default-features = false, features = ["zlib"] } +tar = { version = "0.4.35", default-features = false } +url = "2.2.2" + +[[bench]] +name = "resolve" +harness = false diff --git a/benches/benchsuite/benches/resolve.rs b/benches/benchsuite/benches/resolve.rs new file mode 100644 index 00000000000..266c9c93a9c --- /dev/null +++ b/benches/benchsuite/benches/resolve.rs @@ -0,0 +1,327 @@ +use cargo::core::compiler::{CompileKind, RustcTargetData}; +use cargo::core::resolver::features::{CliFeatures, FeatureOpts, FeatureResolver, ForceAllTargets}; +use cargo::core::resolver::{HasDevUnits, ResolveBehavior}; +use cargo::core::{PackageIdSpec, Workspace}; +use cargo::ops::WorkspaceResolve; +use cargo::Config; +use criterion::{criterion_group, criterion_main, Criterion}; +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::Command; +use url::Url; + +// This is an arbitrary commit that existed when I started. This helps +// ensure consistent results. It can be updated if needed, but that can +// make it harder to compare results with older versions of cargo. +const CRATES_IO_COMMIT: &str = "85f7bfd61ea4fee08ec68c468762e886b2aebec6"; + +fn setup() { + create_home(); + create_target_dir(); + clone_index(); + unpack_workspaces(); +} + +fn root() -> PathBuf { + let mut p = PathBuf::from(env!("CARGO_TARGET_TMPDIR")); + p.push("bench"); + p +} + +fn target_dir() -> PathBuf { + let mut p = root(); + p.push("target"); + p +} + +fn cargo_home() -> PathBuf { + let mut p = root(); + p.push("chome"); + p +} + +fn index() -> PathBuf { + let mut p = root(); + p.push("index"); + p +} + +fn workspaces_path() -> PathBuf { + let mut p = root(); + p.push("workspaces"); + p +} + +fn registry_url() -> Url { + Url::from_file_path(index()).unwrap() +} + +fn create_home() { + let home = cargo_home(); + if !home.exists() { + fs::create_dir_all(&home).unwrap(); + } + fs::write( + home.join("config.toml"), + format!( + r#" + [source.crates-io] + replace-with = 'local-snapshot' + + [source.local-snapshot] + registry = '{}' + "#, + registry_url() + ), + ) + .unwrap(); +} + +fn create_target_dir() { + // This is necessary to ensure the .rustc_info.json file is written. + // Otherwise it won't be written, and it is very expensive to create. + if !target_dir().exists() { + std::fs::create_dir_all(target_dir()).unwrap(); + } +} + +/// This clones crates.io at a specific point in time into tmp/index. +fn clone_index() { + let index = index(); + let maybe_git = |command: &str| { + let status = Command::new("git") + .current_dir(&index) + .args(command.split_whitespace().collect::>()) + .status() + .expect("git should be installed"); + status.success() + }; + let git = |command: &str| { + if !maybe_git(command) { + panic!("failed to run git command: {}", command); + } + }; + if index.exists() { + if maybe_git(&format!( + "rev-parse -q --verify {}^{{commit}}", + CRATES_IO_COMMIT + )) { + // Already fetched. + return; + } + } else { + fs::create_dir_all(&index).unwrap(); + git("init --bare"); + git("remote add origin https://github.com/rust-lang/crates.io-index"); + } + git(&format!("fetch origin {}", CRATES_IO_COMMIT)); + git("branch -f master FETCH_HEAD"); +} + +/// This unpacks the compressed workspace skeletons into tmp/workspaces. +fn unpack_workspaces() { + let ws_dir = Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .unwrap() + .join("workspaces"); + let archives = fs::read_dir(ws_dir) + .unwrap() + .map(|e| e.unwrap().path()) + .filter(|p| p.extension() == Some(std::ffi::OsStr::new("tgz"))); + for archive in archives { + let name = archive.file_stem().unwrap(); + let f = fs::File::open(&archive).unwrap(); + let f = flate2::read::GzDecoder::new(f); + let dest = workspaces_path().join(&name); + if dest.exists() { + fs::remove_dir_all(&dest).unwrap(); + } + let mut archive = tar::Archive::new(f); + archive.unpack(workspaces_path()).unwrap(); + } +} + +struct ResolveInfo<'cfg> { + ws: Workspace<'cfg>, + requested_kinds: [CompileKind; 1], + target_data: RustcTargetData<'cfg>, + cli_features: CliFeatures, + specs: Vec, + has_dev_units: HasDevUnits, + force_all_targets: ForceAllTargets, + ws_resolve: WorkspaceResolve<'cfg>, +} + +/// Vec of `(ws_name, ws_root)`. +fn workspaces() -> Vec<(String, PathBuf)> { + // CARGO_BENCH_WORKSPACES can be used to override, otherwise it just uses + // the workspaces in the workspaces directory. + let mut ps: Vec<_> = match std::env::var_os("CARGO_BENCH_WORKSPACES") { + Some(s) => std::env::split_paths(&s).collect(), + None => fs::read_dir(workspaces_path()) + .unwrap() + .map(|e| e.unwrap().path()) + // These currently fail in most cases on Windows due to long + // filenames in the git checkouts. + .filter(|p| { + !(cfg!(windows) + && matches!(p.file_name().unwrap().to_str().unwrap(), "servo" | "tikv")) + }) + .collect(), + }; + // Sort so it is consistent. + ps.sort(); + ps.into_iter() + .map(|p| (p.file_name().unwrap().to_str().unwrap().to_owned(), p)) + .collect() +} + +/// Helper for resolving a workspace. This will run the resolver once to +/// download everything, and returns all the data structures that are used +/// during resolution. +fn do_resolve<'cfg>(config: &'cfg Config, ws_root: &Path) -> ResolveInfo<'cfg> { + let requested_kinds = [CompileKind::Host]; + let ws = cargo::core::Workspace::new(&ws_root.join("Cargo.toml"), config).unwrap(); + let target_data = RustcTargetData::new(&ws, &requested_kinds).unwrap(); + let cli_features = CliFeatures::from_command_line(&[], false, true).unwrap(); + let pkgs = cargo::ops::Packages::Default; + let specs = pkgs.to_package_id_specs(&ws).unwrap(); + let has_dev_units = HasDevUnits::Yes; + let force_all_targets = ForceAllTargets::No; + // Do an initial run to download anything necessary so that it does + // not confuse criterion's warmup. + let ws_resolve = cargo::ops::resolve_ws_with_opts( + &ws, + &target_data, + &requested_kinds, + &cli_features, + &specs, + has_dev_units, + force_all_targets, + ) + .unwrap(); + ResolveInfo { + ws, + requested_kinds, + target_data, + cli_features, + specs, + has_dev_units, + force_all_targets, + ws_resolve, + } +} + +/// Creates a new Config. +/// +/// This is separate from `do_resolve` to deal with the ownership and lifetime. +fn make_config(ws_root: &Path) -> Config { + let shell = cargo::core::Shell::new(); + let mut config = cargo::util::Config::new(shell, ws_root.to_path_buf(), cargo_home()); + // Configure is needed to set the target_dir which is needed to write + // the .rustc_info.json file which is very expensive. + config + .configure( + 0, + false, + None, + false, + false, + false, + &Some(target_dir()), + &[], + &[], + ) + .unwrap(); + config +} + +/// Benchmark of the full `resovle_ws_with_opts` which runs the resolver +/// twice, the feature resolver, and more. This is a major component of a +/// regular cargo build. +fn resolve_ws(c: &mut Criterion) { + setup(); + let mut group = c.benchmark_group("resolve_ws"); + for (ws_name, ws_root) in workspaces() { + let config = make_config(&ws_root); + // The resolver info is initialized only once in a lazy fashion. This + // allows criterion to skip this workspace if the user passes a filter + // on the command-line (like `cargo bench -- resolve_ws/tikv`). + // + // Due to the way criterion works, it tends to only run the inner + // iterator once, and we don't want to call `do_resolve` in every + // "step", since that would just be some useless work. + let mut lazy_info = None; + group.bench_function(&ws_name, |b| { + let ResolveInfo { + ws, + requested_kinds, + target_data, + cli_features, + specs, + has_dev_units, + force_all_targets, + .. + } = lazy_info.get_or_insert_with(|| do_resolve(&config, &ws_root)); + b.iter(|| { + cargo::ops::resolve_ws_with_opts( + ws, + target_data, + requested_kinds, + cli_features, + specs, + *has_dev_units, + *force_all_targets, + ) + .unwrap(); + }) + }); + } + group.finish(); +} + +/// Benchmark of the feature resolver. +fn feature_resolver(c: &mut Criterion) { + setup(); + let mut group = c.benchmark_group("feature_resolver"); + for (ws_name, ws_root) in workspaces() { + let config = make_config(&ws_root); + let mut lazy_info = None; + group.bench_function(&ws_name, |b| { + let ResolveInfo { + ws, + requested_kinds, + target_data, + cli_features, + specs, + has_dev_units, + ws_resolve, + .. + } = lazy_info.get_or_insert_with(|| do_resolve(&config, &ws_root)); + b.iter(|| { + let feature_opts = FeatureOpts::new_behavior(ResolveBehavior::V2, *has_dev_units); + FeatureResolver::resolve( + ws, + target_data, + &ws_resolve.targeted_resolve, + &ws_resolve.pkg_set, + cli_features, + specs, + requested_kinds, + feature_opts, + ) + .unwrap(); + }) + }); + } + group.finish(); +} + +// Criterion complains about the measurement time being too small, but the +// measurement time doesn't seem important to me, what is more important is +// the number of iterations which defaults to 100, which seems like a +// reasonable default. Otherwise, the measurement time would need to be +// changed per workspace. We wouldn't want to spend 60s on every workspace, +// that would take too long and isn't necessary for the smaller workspaces. +criterion_group!(benches, resolve_ws, feature_resolver); +criterion_main!(benches); diff --git a/benches/capture/Cargo.toml b/benches/capture/Cargo.toml new file mode 100644 index 00000000000..9f529a57f9c --- /dev/null +++ b/benches/capture/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "capture" +version = "0.1.0" +edition = "2018" +license = "MIT OR Apache-2.0" +description = "Tool for capturing a real-world workspace for benchmarking." + +[dependencies] +cargo_metadata = "0.14.0" +flate2 = { version = "1.0.3", default-features = false, features = ["zlib"] } +tar = { version = "0.4.35", default-features = false } +toml = "0.5.8" diff --git a/benches/capture/src/main.rs b/benches/capture/src/main.rs new file mode 100644 index 00000000000..f6f02c4ba05 --- /dev/null +++ b/benches/capture/src/main.rs @@ -0,0 +1,164 @@ +//! This tool helps to capture the `Cargo.toml` files of a workspace. +//! +//! Run it by passing a list of workspaces to capture. +//! Use the `-f` flag to allow it to overwrite existing captures. +//! The workspace will be saved in a `.tgz` file in the `../workspaces` directory. + +use flate2::{Compression, GzBuilder}; +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::Command; + +fn main() { + let force = std::env::args().any(|arg| arg == "-f"); + let dest = Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .unwrap() + .join("workspaces"); + if !dest.exists() { + panic!("expected {} to exist", dest.display()); + } + for arg in std::env::args().skip(1).filter(|arg| !arg.starts_with("-")) { + let source_root = fs::canonicalize(arg).unwrap(); + capture(&source_root, &dest, force); + } +} + +fn capture(source_root: &Path, dest: &Path, force: bool) { + let name = Path::new(source_root.file_name().unwrap()); + let mut dest_gz = PathBuf::from(dest); + dest_gz.push(name); + dest_gz.set_extension("tgz"); + if dest_gz.exists() { + if !force { + panic!( + "dest {:?} already exists, use -f to force overwriting", + dest_gz + ); + } + fs::remove_file(&dest_gz).unwrap(); + } + let vcs_info = capture_vcs_info(source_root, force); + let dst = fs::File::create(&dest_gz).unwrap(); + let encoder = GzBuilder::new() + .filename(format!("{}.tar", name.to_str().unwrap())) + .write(dst, Compression::best()); + let mut ar = tar::Builder::new(encoder); + ar.mode(tar::HeaderMode::Deterministic); + if let Some(info) = &vcs_info { + add_ar_file(&mut ar, &name.join(".cargo_vcs_info.json"), info); + } + + // Gather all local packages. + let metadata = cargo_metadata::MetadataCommand::new() + .manifest_path(source_root.join("Cargo.toml")) + .features(cargo_metadata::CargoOpt::AllFeatures) + .exec() + .expect("cargo_metadata failed"); + let mut found_root = false; + for package in &metadata.packages { + if package.source.is_some() { + continue; + } + let manifest_path = package.manifest_path.as_std_path(); + copy_manifest(&manifest_path, &mut ar, name, &source_root); + found_root |= manifest_path == source_root.join("Cargo.toml"); + } + if !found_root { + // A virtual workspace. + let contents = fs::read_to_string(source_root.join("Cargo.toml")).unwrap(); + assert!(!contents.contains("[package]")); + add_ar_file(&mut ar, &name.join("Cargo.toml"), &contents); + } + let lock = fs::read_to_string(source_root.join("Cargo.lock")).unwrap(); + add_ar_file(&mut ar, &name.join("Cargo.lock"), &lock); + let encoder = ar.into_inner().unwrap(); + encoder.finish().unwrap(); + eprintln!("created {}", dest_gz.display()); +} + +fn copy_manifest( + manifest_path: &Path, + ar: &mut tar::Builder, + name: &Path, + source_root: &Path, +) { + let relative_path = manifest_path + .parent() + .unwrap() + .strip_prefix(source_root) + .expect("workspace member should be under workspace root"); + let relative_path = name.join(relative_path); + let contents = fs::read_to_string(&manifest_path).unwrap(); + let mut manifest: toml::Value = toml::from_str(&contents).unwrap(); + let remove = |obj: &mut toml::Value, name| { + let table = obj.as_table_mut().unwrap(); + if table.contains_key(name) { + table.remove(name); + } + }; + remove(&mut manifest, "lib"); + remove(&mut manifest, "bin"); + remove(&mut manifest, "example"); + remove(&mut manifest, "test"); + remove(&mut manifest, "bench"); + remove(&mut manifest, "profile"); + if let Some(package) = manifest.get_mut("package") { + remove(package, "default-run"); + } + let contents = toml::to_string(&manifest).unwrap(); + add_ar_file(ar, &relative_path.join("Cargo.toml"), &contents); + add_ar_file(ar, &relative_path.join("src").join("lib.rs"), ""); +} + +fn add_ar_file(ar: &mut tar::Builder, path: &Path, contents: &str) { + let mut header = tar::Header::new_gnu(); + header.set_entry_type(tar::EntryType::file()); + header.set_mode(0o644); + header.set_size(contents.len() as u64); + header.set_mtime(123456789); + header.set_cksum(); + ar.append_data(&mut header, path, contents.as_bytes()) + .unwrap(); +} + +fn capture_vcs_info(ws_root: &Path, force: bool) -> Option { + let maybe_git = |command: &str| { + Command::new("git") + .current_dir(ws_root) + .args(command.split_whitespace().collect::>()) + .output() + .expect("git should be installed") + }; + assert!(ws_root.join("Cargo.toml").exists()); + let relative = maybe_git("ls-files --full-name Cargo.toml"); + if !relative.status.success() { + if !force { + panic!("git repository not detected, use -f to force"); + } + return None; + } + let p = Path::new(std::str::from_utf8(&relative.stdout).unwrap().trim()); + let relative = p.parent().unwrap(); + if !force { + let has_changes = !maybe_git("diff-index --quiet HEAD .").status.success(); + if has_changes { + panic!("git repo appears to have changes, use -f to force, or clean the repo"); + } + } + let commit = maybe_git("rev-parse HEAD"); + assert!(commit.status.success()); + let commit = std::str::from_utf8(&commit.stdout).unwrap().trim(); + let remote = maybe_git("remote get-url origin"); + assert!(remote.status.success()); + let remote = std::str::from_utf8(&remote.stdout).unwrap().trim(); + let info = format!( + "{{\n \"git\": {{\n \"sha1\": \"{}\",\n \"remote\": \"{}\"\n }},\ + \n \"path_in_vcs\": \"{}\"\n}}\n", + commit, + remote, + relative.display() + ); + eprintln!("recording vcs info:\n{}", info); + Some(info) +} diff --git a/benches/workspaces/cargo.tgz b/benches/workspaces/cargo.tgz new file mode 100644 index 00000000000..653aff982d6 Binary files /dev/null and b/benches/workspaces/cargo.tgz differ diff --git a/benches/workspaces/diem.tgz b/benches/workspaces/diem.tgz new file mode 100644 index 00000000000..e047c6cd04b Binary files /dev/null and b/benches/workspaces/diem.tgz differ diff --git a/benches/workspaces/empty.tgz b/benches/workspaces/empty.tgz new file mode 100644 index 00000000000..1a7d555b407 Binary files /dev/null and b/benches/workspaces/empty.tgz differ diff --git a/benches/workspaces/gecko-dev.tgz b/benches/workspaces/gecko-dev.tgz new file mode 100644 index 00000000000..e89c676b2b2 Binary files /dev/null and b/benches/workspaces/gecko-dev.tgz differ diff --git a/benches/workspaces/rust.tgz b/benches/workspaces/rust.tgz new file mode 100644 index 00000000000..74da4759b02 Binary files /dev/null and b/benches/workspaces/rust.tgz differ diff --git a/benches/workspaces/servo.tgz b/benches/workspaces/servo.tgz new file mode 100644 index 00000000000..51116436979 Binary files /dev/null and b/benches/workspaces/servo.tgz differ diff --git a/benches/workspaces/substrate.tgz b/benches/workspaces/substrate.tgz new file mode 100644 index 00000000000..81c3874f655 Binary files /dev/null and b/benches/workspaces/substrate.tgz differ diff --git a/benches/workspaces/tikv.tgz b/benches/workspaces/tikv.tgz new file mode 100644 index 00000000000..74add19b38a Binary files /dev/null and b/benches/workspaces/tikv.tgz differ diff --git a/benches/workspaces/toml-rs.tgz b/benches/workspaces/toml-rs.tgz new file mode 100644 index 00000000000..9acab198240 Binary files /dev/null and b/benches/workspaces/toml-rs.tgz differ diff --git a/src/cargo/ops/mod.rs b/src/cargo/ops/mod.rs index 50143978f22..e81486f3dd5 100644 --- a/src/cargo/ops/mod.rs +++ b/src/cargo/ops/mod.rs @@ -28,6 +28,7 @@ pub use self::registry::{needs_custom_http_transport, registry_login, registry_l pub use self::registry::{publish, registry_configuration, RegistryConfig}; pub use self::resolve::{ add_overrides, get_resolved_packages, resolve_with_previous, resolve_ws, resolve_ws_with_opts, + WorkspaceResolve, }; pub use self::vendor::{vendor, VendorOptions}; diff --git a/src/doc/contrib/src/SUMMARY.md b/src/doc/contrib/src/SUMMARY.md index 4a48c03070e..0d63295ffc8 100644 --- a/src/doc/contrib/src/SUMMARY.md +++ b/src/doc/contrib/src/SUMMARY.md @@ -16,5 +16,5 @@ - [Tests](./tests/index.md) - [Running Tests](./tests/running.md) - [Writing Tests](./tests/writing.md) - - [Profiling](./tests/profiling.md) + - [Benchmarking and Profiling](./tests/profiling.md) - [Design Principles](./design.md) diff --git a/src/doc/contrib/src/tests/profiling.md b/src/doc/contrib/src/tests/profiling.md index a95d60fbce3..1cc980ca3e3 100644 --- a/src/doc/contrib/src/tests/profiling.md +++ b/src/doc/contrib/src/tests/profiling.md @@ -1,4 +1,4 @@ -# Profiling +# Benchmarking and Profiling ## Internal profiler @@ -11,7 +11,15 @@ profile stack to print results for. CARGO_PROFILE=3 cargo generate-lockfile ``` -## Informal profiling +## Benchmarking + +### Benchsuite + +Head over to the [`benches` +directory](https://github.com/rust-lang/cargo/tree/master/benches) for more +information about the benchmarking suite. + +### Informal benchmarking The overhead for starting a build should be kept as low as possible (preferably, well under 0.5 seconds on most projects and systems). Currently, @@ -23,12 +31,10 @@ the primary parts that affect this are: * Scanning the local project. * Building the unit dependency graph. -We currently don't have any automated systems or tools for measuring or -tracking the startup time. We informally measure these on changes that are -likely to affect the performance. Usually this is done by measuring the time -for `cargo build` to finish in a large project where the build is fresh (no -actual compilation is performed). [Hyperfine] is a command-line tool that can -be used to roughly measure the difference between different commands and -settings. +One way to test this is to use [hyperfine]. This is a tool that can be used to +measure the difference between different commands and settings. Usually this +is done by measuring the time it takes for `cargo build` to finish in a large +project where the build is fresh (no actual compilation is performed). Just +run `cargo build` once before using hyperfine. -[Hyperfine]: https://github.com/sharkdp/hyperfine +[hyperfine]: https://github.com/sharkdp/hyperfine