Ability to generate test cases from file path

[ajeetdsouza]

Hey, thanks for creating this! I think a feature to generate tests for each file matching a certain glob would be great.

An existing crate that does this is test-generator, but building something similar here would add a lot of value. For example, a user would be able to create a matrix test where the first parameter is every path matching a certain glob, and the second parameter is configurable via the #[values()] attribute.

[la10736]

Thanks for idea! I thought about that in the past but I didn't find a way to do it without ask to the user to change/introduce build.rs script. So, I'll take another shot to this and take a look to the crate that you pointed to see if it use some good idea that not involves build.rs script. Il lun 12 set 2022, 16:22 ajeetdsouza ***@***.***> ha scritto:

…

Hey, thanks for creating this! I think a feature to generate tests for each file matching a certain glob would be great. An existing crate that does this is test-generator <https://github.com/frehberg/test-generator>, but building something similar here would add a lot of value. For example, a user would be able to create a matrix test where the first parameter is every path matching a certain glob, and the second parameter is configurable via the #[values()] attribute. — Reply to this email directly, view it on GitHub <#163>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5Y34IT34C6Q6MHKUTQNWDV544CNANCNFSM6AAAAAAQKQWHFQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jpmckinney]

As far as I can tell, test-generator does need a build.rs script (unless it's optional?).

[la10736]

Unfortunately build script is not optional 😢 . As far as I know it is not possible to access to something else than the annotated block from procedural macro.

[GeeWee]

Seems like you should be able to access the filesystem? - however getting the path resolution to work might be tricky. I haven't tried this myself though.

[ForsakenHarmony]

the crate testing does it without a build script
https://rustdoc.swc.rs/testing/attr.fixture.html

include_dir also does something similar

[la10736]

Ok, I see the trick

    let base_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR").expect(
        "#[fixture] requires CARGO_MANIFEST_DIR because it's relative to cargo manifest directory",
    ));

It seams feasible to use this trick to generate a value list.

Thanks very much to point it.... I don't have too much time to work on it: a PR would be really appreciated 😄

[jpmckinney]

Here's the relevant file for the #[fixture("glob")] macro in the testing crate: https://github.com/swc-project/swc/blob/main/crates/testing_macros/src/lib.rs

[la10736]

I've already started to implement it but I can work on it just 2 hours for week ... and not every week 😢

Is quite simple to implement the base implementation for the following example

#[rstest]
fn test(#[files("glob_path")] path: PathBuf) {
    ... generate a test for each file 
}

Later I'll extend the syntax to handle

• sequence of glob
• #[exclude("glob_path_to exclude")]

[la10736]

Ok, I completed the first implementation sketch
https://github.com/la10736/rstest/tree/values_from_files

In playground crate you'll find an example of how to use it. If somebody can try it would be really appreciated. THX

[ju1ius]

@la10736 Hi, this is a great step forward ! 👍🏻

However this does not cover the case where there can be several test cases per file.

For example, let's say we're writing an HTML parser and we want to use the official Web Platform Tests repository for that (html5lib-tests, which provides language-agnostic machine-readable test cases).

In this case we'd need an attribute accepting a glob and a mapping function type Mapper<T> = Fn(&Path) -> Vec<T> which would allow calling an #[rstest]-attributed function once per T:

#[rstest]
// The name is probably not right but you get the idea
#[files_mapped_by("tests/**/*.test", map_file)]
fn some_test(case: Testcase) {
  assert_passes!(case)
}

struct TestCase { /* ... */ }

fn map_file(path: &Path) -> Vec<TestCase> {
  todo!()
}

I don't know if it's even possible in current Rust but I'm mentioning it because I've been in this situation a few times and so far I haven't been able to find a library that can handle this.

[la10736]

I already thought about this. My Idea is to try to implement something like this but is not trivial because the deserialization code should be called by procedural macro....

[la10736]

I decided to close this ticket as completed. I'll open a new ones for creating tests from json, csv, yaml and xml (do somebody use it in a nowadays?) files.
@ju1ius I took a look to html5lib-test files and the only Idea that I have is that the map function should be an external script/application that return a serialized list of test (bytes or strings): we can think about something to compile it at compile time but really and I don't love this kind of solutions.