bin not found causes "ignored" not an error

[sourcefrog]

In trying to set up trycmd I had this md file:

```trycmd
$ cargo mutants --help
```

This gives a result of ignored.

With --nocapture I could find out:

running 1 test
[              trycmd::runner] 	Substitutions { vars: {"[CWD]": "/home/mbp/src/mutants"}, unused: {"[EXE]"} }
[              trycmd::runner] 	bin="cargo" not found
[              trycmd::runner] 	Case: Ok(
    Output {
        path: "trycmd/hello.trycmd",
        id: Some(
            "4",
        ),
        spawn: Spawn {
            exit: None,
            status: Skipped,
        },
        stdout: None,
        stderr: None,
        fs: Filesystem {
            context: [],
        },
    },
)
Testing trycmd/hello.trycmd:4 ... ignored
test trycmd_tests ... ok

It seems to me that "file not found" should probably be a test failure (by default?) not ignored?

[epage]

The idea was to gracefully allow mixing untestable examples with testable examples.

We do have the ignore tag and it is unlikely people will have testable and untestable examples within the same code block, so we could probably change this.

[sourcefrog]

Yep, having an option to ignore seems fine, but maybe it should be the wrong default.... Martin

…

On Wed, Jul 20, 2022 at 7:58 AM Ed Page ***@***.***> wrote: The idea was to gracefully allow mixing untestable examples with testable examples. We do have the ignore tag and it is unlikely people will have testable and untestable examples within the same code block, so we could probably change this. — Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACUR43O2DSIUXTPBVFN2ALVVAHZPANCNFSM5Z4NPIBQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[kleinesfilmroellchen]

It seems like I had the exact error, just that I couldn't even access detailed error information via --nocapture. This is definitely confusing as I first believed it to be a platform-specific error.

[epage]

0.14.0 is released with this changed.

[epage]

I had forgotten: this is important for when examples are conditionally present: clap-rs/clap#4439

[kleinesfilmroellchen]

Interesting; is there a better solution that covers both use cases?

[AucaCoyan]

Hi! I'm a bit new to the project. What do you thing about returning an error if a binary is either skipped or doesn't have a name?
The ignored keyword tells my common sense that I wrote a #[skip] on purpose or I forgot to include in a glob path like tests/**/*.toml.
Having a console output "test is ignored" but not telling me why was a hard thing to debug in my case

[epage]

I'd recommend reviewing the thread for what has been previously considered.