Line coverage mismatch on trybuild-based proc-macros tests

[dbarbosapn]

Hello,

I'm seeing some big discrepancies in code coverage when testing proc macros with trybuild.

Crate structure:
├── src
│ ├── lib.rs
├── tests
│ ├── testcases
│ │ ├── fail
│ │ │ ├── will_fail.rs
│ │ │ ├── will_fail.stderr
│ │ ├── pass
│ │ │ ├── will_pass.rs
│ ├── tests.rs

The content of tests.rs is:

#[cfg(test)]
mod macro_tests {
    #[test]
    fn macro_failures() {
        let t = trybuild::TestCases::new();
        t.compile_fail("tests/testcases/fail/*.rs");
    }

    #[test]
    fn macro_successes() {
        let t = trybuild::TestCases::new();
        t.pass("tests/testcases/pass/*.rs");
    }
}

The content of lib.rs is:

// Copyright (c) Microsoft Corporation. All Rights Reserved.

use proc_macro::TokenStream;
use quote::quote;

/// Marks async function to be executed by the mycrate runtime.
///
/// ## Usage
///
/// ```ignore
/// #[mycrate::rt::main]
/// async fn main() {
///     println!("Hello world");
/// }
/// ```
#[proc_macro_attribute]
pub fn rt_main(_: TokenStream, item: TokenStream) -> TokenStream {
    let mut input = syn::parse_macro_input!(item as syn::ItemFn);
    let attrs = &input.attrs;
    let vis = &input.vis;
    let sig = &mut input.sig;
    let body = &input.block;
    let name = &sig.ident;

    if sig.asyncness.is_none() {
        return syn::Error::new_spanned(
            sig.fn_token,
            "function must be async to use the attribute",
        )
        .to_compile_error()
        .into();
    }

    sig.asyncness = None;

    (quote! {
        #(#attrs)*
        #vis #sig {
            mycrate::rt::System::new(stringify!(#name))
                .block_on(async move { #body })
        }
    })
    .into()
}

/// Marks async test function to be executed by the mycrate runtime.
///
/// ## Usage
///
/// ```ignore
/// #[mycrate::rt::test]
/// async fn my_test() {
///     assert!(true);
/// }
/// ```
#[proc_macro_attribute]
pub fn rt_test(_: TokenStream, item: TokenStream) -> TokenStream {
    let input = syn::parse_macro_input!(item as syn::ItemFn);

    let ret = &input.sig.output;
    let name = &input.sig.ident;
    let body = &input.block;
    let attrs = &input.attrs;
    let mut has_test_attr = false;

    for attr in attrs {
        if attr.path().is_ident("test") {
            has_test_attr = true;
        }
    }

    if input.sig.asyncness.is_none() {
        return syn::Error::new_spanned(
            input.sig.fn_token,
            format!(
                "function must be async to use the attribute, {}",
                input.sig.ident
            ),
        )
        .to_compile_error()
        .into();
    }

    let result = if has_test_attr {
        quote! {
            #(#attrs)*
            fn #name() #ret {
                mycrate::rt::System::new("test")
                    .block_on(async { #body })
            }
        }
    } else {
        quote! {
            #[test]
            #(#attrs)*
            fn #name() #ret {
                mycrate::rt::System::new("test")
                    .block_on(async { #body })
            }
        }
    };

    result.into()
}

This is what workspace coverage is reporting:

But, when the file is clicked, all lines and regions are covered.

This is affecting our CI gates so we'll have to exclude macros from coverage, but would love to get some help on this.

Additional details:

• We're using latest cargo-llvm-cov.
• When running with --show-missing-lines, it reports the same coverage issue, but no lines are reported as uncovered.
• After adding the macros tests, we noticed "warning: 3 functions have mismatched data" popped up.

Thanks in advance!

[taiki-e]

But, when the file is clicked, all lines and regions are covered.

I have not yet checked the reproduction locally, but I suspect this is the same problem as #324.

EDIT: I will not be able to reproduce the problem locally as a complete reproduction is not provided in the first place.

[taiki-e]

I suspect this is the same problem as #324.

Ah, nah, maybe another issue is involved because the line coverage is also affected.

In any case, I think we need a complete reproduction, including what is tested in trybuild.

[PollRobots]

I uploaded https://github.com/PollRobots/min-repo

If I run

cargo llvm-cov --open

I can see that all tests run and succeed, including trybuild tests that verify each error branch in the macro.
But in the coverage report the error branches are clearly labeled as not taken

[PollRobots]

@taiki-e hi, is this something that you will have time to look at?

[PollRobots]

@taiki-e --- this has a complete and minimal repo that demonstrates the issue. Is there more that I can provide?

[taiki-e]

Thanks for the repro. This seems to be an issue is due to trybuild 1.0.97 regression (dtolnay/trybuild#283).

trybuild = "=1.0.96" # work as expected
trybuild = "=1.0.97" # not work as expected