No documentation on exit code 5

[amueller]

pytest fails if no tests were run and returns exit code 5.
I can't seem to find any documentation on this on the website.

I had students run pytest on travis, and CI failed because of the non-zero exit code, but they couldn't figure out what was going on. I think it would be great if either or both of the following happened:

a) The error message says it's failing because there are no tests. Currently it says "0 tests run" but it's unclear that this would cause a failure.

b) there is some documentation online that explains what "The command "py.test" exited with 5" means.

[nicoddemus]

Hi @amueller, thanks for taking the time to write this.

I think it would be a good addition to the docs to document the possible return codes, AFAIK they are not documented currently. We would be happy to merge an addition to the docs, would you like to contribute with a PR?

Just to confirm, the message no tests run appears in yellow in the console right (that's in pytest 3.0+)? TBH that seems to me enough to figure out something is amiss from my point of view.

[amueller]

Thanks for your reply.

@nicoddemus The message was enough for me to figure out what was going on, but multiple of my students failed to do so. I think it's a bit implicit that that is what makes travis fail, but I don't have a strong opinion.

I don't have time to add to the docs right now (my unread github notifications are at 5000 right now).
Sorry for the drive-by.

[mgriffin-skimlinks]

By coincidence I discovered this behavior today too, but my take is that this is a bug. Surely "all tests passed" has an exit code of 0 even if there aren't any tests?

[The-Compiler]

@mgriffin-skimlinks It isn't, it was deliberately introduced. See #500, #662, #812, #817.

[pawelad]

Pull request: #2320

[amueller]

thanks @pawelad :)