bin not found causes "ignored" not an error

[sourcefrog]

In trying to set up trycmd I had this md file:

```trycmd
$ cargo mutants --help
```

This gives a result of ignored.

With --nocapture I could find out:

running 1 test
[              trycmd::runner] 	Substitutions { vars: {"[CWD]": "/home/mbp/src/mutants"}, unused: {"[EXE]"} }
[              trycmd::runner] 	bin="cargo" not found
[              trycmd::runner] 	Case: Ok(
    Output {
        path: "trycmd/hello.trycmd",
        id: Some(
            "4",
        ),
        spawn: Spawn {
            exit: None,
            status: Skipped,
        },
        stdout: None,
        stderr: None,
        fs: Filesystem {
            context: [],
        },
    },
)
Testing trycmd/hello.trycmd:4 ... ignored
test trycmd_tests ... ok

It seems to me that "file not found" should probably be a test failure (by default?) not ignored?

[epage]

The idea was to gracefully allow mixing untestable examples with testable examples.

We do have the ignore tag and it is unlikely people will have testable and untestable examples within the same code block, so we could probably change this.

[sourcefrog]

Yep, having an option to ignore seems fine, but maybe it should be the wrong default.... Martin

…

On Wed, Jul 20, 2022 at 7:58 AM Ed Page ***@***.***> wrote: The idea was to gracefully allow mixing untestable examples with testable examples. We do have the ignore tag and it is unlikely people will have testable and untestable examples within the same code block, so we could probably change this. — Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACUR43O2DSIUXTPBVFN2ALVVAHZPANCNFSM5Z4NPIBQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[kleinesfilmroellchen]

It seems like I had the exact error, just that I couldn't even access detailed error information via --nocapture. This is definitely confusing as I first believed it to be a platform-specific error.

[epage]

0.14.0 is released with this changed.

[epage]

I had forgotten: this is important for when examples are conditionally present: clap-rs/clap#4439

[kleinesfilmroellchen]

Interesting; is there a better solution that covers both use cases?

[AucaCoyan]

Hi! I'm a bit new to the project. What do you thing about returning an error if a binary is either skipped or doesn't have a name?
The ignored keyword tells my common sense that I wrote a #[skip] on purpose or I forgot to include in a glob path like tests/**/*.toml.
Having a console output "test is ignored" but not telling me why was a hard thing to debug in my case

[epage]

I'd recommend reviewing the thread for what has been previously considered.

[epage]

@veeso in #386 proposes making failing an opt-in with TestCases::set_fail_on_unresolved_bin.

This allows for both behaviors to be present and even allows us to delay making this a breaking change.

Based on the current design, a set_ prefix wouldn't be used. I lean towards this being unknown rather than unresolved (as we don't resolve binary names to paths). So an alternative name could be TestCases::fail_on_unknown_bin(bool) or TestCases::fail_unknown_bins(bool)

Alternatives:

• TestCases::unknown_bin( Ignore | Fail )
• TestCases::unknown_bin(Fn(name, args) -> Result)
  • This would be one way to register rust code as a "binary" like for Running "standard" unix commands #168.
  • Hmm, looks like I was concerned about overly broad functions and composability (Running "standard" unix commands #168 (comment))

[veeso]

I've renamed the function/params to fail_on_unknown_bin

[epage]

@veeso what are your thoughts on the alternatives?

[veeso]

I think using an enum with the rule could be good as well and could prevent breaking changes if one day we decide to extend this behaviour to new kinds.

A closure would be too complex imo