mix test --failed errors when failed tests are missing

[Efesto]

When mix test --failed is executed after a failed execution of mix test, instead of just returning "0 tests, 0 failures", returns with a specific error message and exit code 1.

This is not the same when for example the error manifest file is empty or contains files that cannot be found.

The reason for this is to raise cases in which failing tests with a dynamically generated name are ignored when running mix test --failed and calling for more attention when generating test names

For example, when generating test titles containing DB-based fixtures ids:

for country <- countries_from_db do
  test "country #{country.id} does something" do
    # ...test test...
  end
end

when the correct way would be:

for country <- countries_from_db do
  test "country #{country.name} does something" do
    # ...test test...
  end
end

[josevalim]

Thank you but I am not sure if we should go ahead this path. After all, you could have a failed test which is then removed. Similarly, you could rename a test as well and fix it. I don't think we can reliably detect this scenario. I am not sure if there is a better answer than: make sure your test names are deterministic (and I'd say having non-deterministic test names is extremely rare).

[Efesto]

Thank you but I am not sure if we should go ahead this path. After all, you could have a failed test which is then removed. Similarly, you could rename a test as well and fix it. I don't think we can reliably detect this scenario. I am not sure if there is a better answer than: make sure your test names are deterministic (and I'd say having non-deterministic test names is extremely rare).

I think the most common scenario when a test manifest file contains tests that don't exist anymore is either because the manifest is old and the tests changed or indeed the names are ever-changing, and this would help raising a flag on something wrong in the tests themselves rather than silently accepting.
This applies to CIs in particular but also in local development where devs have more awareness of how to use mix test --failed

In the case I've encountered we have a flakey tests detector that uses mix test --failed for re-executing failed test and determine if they are indeed flakey, after x time collect them and report. We had a set of such tests that were constantly failing and passed to master.

[josevalim]

The change branch scenario is a good one. I don't think we can avoid from having either false positives or false negatives but I still believe this pull request does not fully address the issues you have in mind, because we only warn in the cases none of the tests are found, so if you have 1 test still relevant, all of the issues you describe still exist, and the user is still unaware.

In the scenarios you describe, you should be combining --stale and --failed, especially as --failed is meant for quick development cycles. If you believe we could improve the docs, I'll gladly review that, but I don't think we should go ahead with this behaviour change. thanks!