Standalone tests can silently fail

[awaelchli]

🐛 Bug

The CI passes with a green check mark, but the logs clearly show that the standalone tests fail.

To Reproduce

Can be seen on the latest commits on master.

At least two parametrization of this test fail: test_progress_bar_max_val_check_interval

Expected behavior

The CI fails when a test fails.

Environment

PL master

Additional context

As I always say, don't trust the CI.

cc @tchaton @rohitgr7 @akihironitta @carmocca @Borda

[awaelchli]

@rohitgr7 I'm not 100% sure but it looks like this started to happen since #11657

[Borda]

ok, just checking and we already have set -e which shall enforce upstream the error
https://github.com/PyTorchLightning/pytorch-lightning/blob/2e5728a4841cb1d3b75123ac941cd36f681f9a11/.azure-pipelines/gpu-tests.yml#L112

[rohitgr7]

this is weird. Just checked, test is passing on lightning cluster

[carmocca]

Did you try using the script?

./tests/standalone_tests.sh -k test_progress_bar_max_val_check_interval

and running all standalone tests?

[rohitgr7]

Thanks, @carmocca!! Now it's failing. Let me check the issue.

btw how is this different from running the tests manually?

[carmocca]

It manages calling multiple (or all) standalone tests with a printed report at the end.

[rohitgr7]

okay.. but still trying to understand why it failed when running with ./tests/standalone_tests.sh and not when running manually, although the test had a bug.