#[ignore] at the top only applies to the first case

[nsunderland1]

Given the following code:

fn main() {}

#[cfg(test)]
mod test {
    use rstest::rstest;

    #[ignore]
    #[rstest]
    #[case::is_ignored()]
    #[case::isnt_ignored()]
    #[case::also_isnt_ignored()]
    fn only_ignores_first() {}

    #[rstest]
    #[case::is_ignored()]
    #[case::is_also_ignored()]
    #[ignore]
    fn ignores_all() {}
}

I expect that all test cases will be ignored. Instead, only_ignores_first only sees the first test case ignored:

running 5 tests
test test::ignores_all::case_1_is_ignored ... ignored
test test::ignores_all::case_2_is_also_ignored ... ignored
test test::only_ignores_first::case_1_is_ignored ... ignored
test test::only_ignores_first::case_2_isnt_ignored ... ok
test test::only_ignores_first::case_3_also_isnt_ignored ... ok

rstest version is 0.18.2

[la10736]

You're expecting wrong 😄
https://docs.rs/rstest/latest/rstest/attr.rstest.html#use-specific-case-attributes

Every case collect the attributes that preceding it and all attributes between the last case and function definition is applied to the function itself and so to all tests.

[nsunderland1]

I agree that as documented it's technically correct, but I do think that this:

#[ignore]
#[rstest]
#[case::is_ignored()]
#[case::isnt_ignored()]
fn only_ignores_first() {}

is not obviously the same as this:

#[rstest]
#[ignore]
#[case::is_ignored()]
#[case::isnt_ignored()]
fn only_ignores_first() {}

In the second example, it's very intuitive for the attribute to only apply to the first case. But in the first example, it doesn't look like it should be applied to a specific case at all. When I see an attribute outside the "scope" of the rstest macro like that, I assume it'll be applied broadly across all cases. Unlike in the first example, there isn't really a clear visual connection to the first case.

Here's another way to frame my perspective. Let's say I'm writing a test with a few cases, and I want to ignore the first case. I am almost certainly going to stick the ignore directly above the case, rather than above the rstest macro, because that's the most visually clear thing to do. If we start from the assumption that that's how almost everyone will write this code, then that suggests that a lot of examples with the ignore above the #[rstest] are probably wrong.

Maybe this doesn't warrant a change in behaviour, especially since that could break existing code. But I think it warrants a warning, since there's an easy way to rewrite it to be more clear, and it'll catch cases where people did this by accident.

[la10736]

Ok, you are partially right. Is true that the attributes before #[rstest] should not be applied to the first case only (that's a bug).

To be coherent I should leave them on the top of the expanded code and that will generate an error (you cannot apply #[ignore] to a module). For all custom attributes they are processed before the rstest attribute but ignore is a special one and the compiler doesn't try to expand it.

Anyway I'll add an error message if I found some attributes before the rstest ones.

THX for reporting it.

[la10736]

Ok, I've come back on this and unfortunately is quite impossible to fix it. There is no way to understand if some attribute is before or not to rstest one: the compiler give only the an AST stream with all attributes without the one that you're function is expanding. To understand better the follow case send the same data to rstest procedural macro.

#[ignore]
#[rstest]
#[case::is_ignored()]
#[case::isnt_ignored()]
fn only_ignores_first() {}

#[rstest]
#[ignore]
#[case::is_ignored()]
#[case::isnt_ignored()]
fn only_ignores_first() {}

The only difference from the first and the second case is the span associated to ignore, but also in this case you cannot relay on it because about the rstest attribute the compiler send only the date inside an eventually given tokens in the rstest attribute parameters.

I've tried also to leverage on attributes span but is too flaky and fragile... depends too much about formatting, spaces docs and comments.... I one word is a mess to try to separate the two cases.

😢