Skip to content

Detect cyclic auxiliary #134180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 126 additions & 1 deletion src/tools/compiletest/src/header/auxiliary.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
//! Code for dealing with test directives that request an "auxiliary" crate to
//! be built and made available to the test in some way.

use std::iter;
use std::collections::{HashMap, HashSet};
use std::fs::File;
use std::path::Path;
use std::{fs, io, iter};

use super::{DirectiveLine, iter_header};
use crate::common::Config;
use crate::header::directives::{AUX_BIN, AUX_BUILD, AUX_CODEGEN_BACKEND, AUX_CRATE, PROC_MACRO};

Expand Down Expand Up @@ -63,3 +67,124 @@ fn parse_aux_crate(r: String) -> (String, String) {
parts.next().expect("missing aux-crate value (e.g. log=log.rs)").to_string(),
)
}

/// Return an error if the given directory has cyclic aux.
pub(crate) fn check_cycles(config: &Config, dir: &Path) -> io::Result<()> {
let mut filenames = vec![];
let mut auxiliaries = HashMap::new();

build_graph(config, dir, dir, &mut filenames, &mut auxiliaries)?;

has_cycle(&filenames, &auxiliaries)
}

fn build_graph(
config: &Config,
dir: &Path,
base_dir: &Path,
filenames: &mut Vec<String>,
auxiliaries: &mut HashMap<String, Vec<String>>,
) -> io::Result<()> {
Comment on lines +81 to +87
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: so I thought about this a bit. build_graph during test collection that does directory traversal and file reading and prop parsing is quite expensive (especially on slower filesystems like native Windows), and I don't think this is necessarily the right place to do the check.

I would've expected that compiletest does something like:

  1. First, it collects the tests based on the path filter (or was it test name filter) that was passed from bootstrap.
  2. Then, compiletest will do some "early props" collection and handling. At this point is when I believe compiletest establishes the basic test conditions and relationships between main test files and their auxiliaries (i.e. if a test will be run or ignored, if it has auxiliaries and such).
  3. Then, compiletest will further collect and handle "late props".

I would've expected that aux cycle detection happens after early prop collection after we establish the relationship between test files and auxiliaries, and not during test discovery and collection.

Alternatively, we could also just declare that we're not going to handle cyclic auxiliaries and "just don't do that", and doing cyclic auxiliaries is considered PEBKAC as "compiletest did exactly what the user instructed it to do".

cc @oli-obk: how does ui_test handle infinitely-recursive auxiliaries?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't: oli-obk/ui_test#123

But it should be fixable now that I have a build manager that handles all of these in one place.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jieyouxu
Thanks for your review.

  1. Then, compiletest will do some "early props" collection and handling. At this point is when I believe compiletest establishes the basic test conditions and relationships between main test files and their auxiliaries (i.e. if a test will be run or ignored, if it has auxiliaries and such).

Now I understood that cycle detection should be run around here, is that right?

FYI: EarlyProps gathers auxiliaries only which are referred from the test file. For example from my manual test files above, cycle1.rs is included in the early_props for aux-cyclic.rs but cycle2.rs is not because only cycle1.rs has a dependency to cycle2.rs. So reading files to make a dependency graph is inevitable, but it should be lighter than reading files by entire directory traversal since we can check the cycle only against test files which contains auxs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question, I'll have to revisit this.

for file in fs::read_dir(dir)? {
let file = file?;
let file_path = file.path();

if file_path.is_dir() {
// explore in sub directory.
build_graph(config, &file_path, base_dir, filenames, auxiliaries)?;
} else {
// We'd like to put a filename with relative path from the auxiliary directory (e.g., ["foo.rs", "foo/bar.rs"]).
let relative_filename = file_path
.strip_prefix(base_dir)
.map_err(|e| io::Error::other(e))?
.to_str()
.unwrap();

filenames.push(relative_filename.to_string());

let mut aux_props = AuxProps::default();
let mut poisoned = false;
let f = File::open(&file_path).expect("open file to parse aux for cycle detection");
iter_header(
config.mode,
&config.suite,
&mut poisoned,
&file_path,
f,
&mut |DirectiveLine { raw_directive: ln, .. }| {
parse_and_update_aux(config, ln, &mut aux_props);
},
);

let mut auxs = vec![];
for aux in aux_props.all_aux_path_strings() {
auxs.push(aux.to_string());
}

if auxs.len() > 0 {
auxiliaries.insert(relative_filename.to_string(), auxs);
}
}
}

Ok(())
}

/// has_cycle checks if the given graph has cycle.
/// It performs with a simple Depth-first search.
fn has_cycle(
filenames: &Vec<String>,
auxiliaries: &HashMap<String, Vec<String>>,
) -> io::Result<()> {
// checked tracks nodes which the function already finished to search.
let mut checked = HashSet::with_capacity(filenames.len());
// During onde DFS exploration, on_search tracks visited nodes.
// If the current node is already in on_search, that's a cycle.
// The capacity `4` is added, because we can guess that an aux dependency is not so a long path.
let mut on_search = HashSet::with_capacity(4);
// path tracks visited nodes in on exploration.
// This is used for generating an error message when a cycle is detected.
let mut path = Vec::with_capacity(4);

for vertex in filenames.iter() {
if !checked.contains(vertex) {
search(filenames, auxiliaries, &vertex, &mut checked, &mut on_search, &mut path)?;
}
}

fn search(
filenames: &Vec<String>,
auxiliaries: &HashMap<String, Vec<String>>,
vertex: &str,
checked: &mut HashSet<String>,
on_search: &mut HashSet<String>,
path: &mut Vec<String>,
) -> io::Result<()> {
if !on_search.insert(vertex.to_string()) {
let mut cyclic_path = vec![vertex];
for v in path.iter().rev() {
if v == vertex {
break;
}
cyclic_path.push(v);
}

return Err(io::Error::other(format!("detect cyclic auxiliary: {:?}", cyclic_path)));
}

if checked.insert(vertex.to_string()) {
path.push(vertex.to_string());
if let Some(auxs) = auxiliaries.get(&vertex.to_string()) {
for aux in auxs.iter() {
search(filenames, auxiliaries, &aux, checked, on_search, path)?;
}
}
path.pop().unwrap();
}

on_search.remove(&vertex.to_string());
Ok(())
}

Ok(())
}
3 changes: 3 additions & 0 deletions src/tools/compiletest/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ use std::{env, fs, vec};

use build_helper::git::{get_git_modified_files, get_git_untracked_files};
use getopts::Options;
use header::auxiliary::check_cycles;
use test::ColorConfig;
use tracing::*;
use walkdir::WalkDir;
Expand Down Expand Up @@ -766,6 +767,8 @@ fn collect_tests_from_dir(
if &file_name != "auxiliary" {
debug!("found directory: {:?}", file_path.display());
collect_tests_from_dir(cx, collector, &file_path, &relative_file_path)?;
} else {
check_cycles(&cx.config, &file_path)?;
}
} else {
debug!("found other file/directory: {:?}", file_path.display());
Expand Down
Loading