test_quantized_models.py times out within Nightly CI

[seemethere]

Currently the nightly pipelines are failing when running pytest . on the test/test_ops.py tests.

Example CircleCI Logs:
https://app.circleci.com/jobs/github/pytorch/vision/85108

Log Excerpt:

+ pytest .
============================= test session starts ==============================
platform linux -- Python 3.6.10, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: $SRC_DIR
collected 286 items

test/test_backbone_utils.py ..                                           [  0%]
test/test_cpp_models.py sssssssssssssssssssssssssssssss                  [ 11%]
test/test_datasets.py ..........                                         [ 15%]
test/test_datasets_samplers.py .....                                     [ 16%]
test/test_datasets_transforms.py ..                                      [ 17%]
test/test_datasets_utils.py ..........                                   [ 20%]
test/test_datasets_video_utils.py ....                                   [ 22%]
test/test_functional_tensor.py ........                                  [ 25%]
test/test_hub.py sss                                                     [ 26%]
test/test_io.py ss............                                           [ 31%]
test/test_models.py .........s.........................................  [ 48%]
test/test_models_detection_utils.py .                                    [ 49%]
test/test_onnx.py sssssssssssss                                          [ 53%]
test/test_ops.py ..ss..ss..ss..ss..ss..ss..ss..ss.s...ss..ss             [ 68%]
Too long with no output (exceeded 10m0s): context deadline exceeded

This is what is currently effecting pytorch/pytorch#33103

Also this may be related to #1528

cc @fmassa

[fmassa]

Thanks for the report!

I think this might be due to @zou3519 changes in pytorch/pytorch#32495, which also caused nested tensors CI to fail, see pytorch/pytorch#33091

[zou3519]

@fmassa does test_ops.py run c++ extensions?

[zou3519]

I don't think pytorch/pytorch#32495 is related because test_ops.py doesn't run c++ extensions directly. However, there's an easy way to test, assuming the CI runs on each PR (send a PR that disables building with ninja). I'll do that and see what happens.

EDIT: Here's another reason why I think pytorch/pytorch#32495 is unrelated. Consider #1850. The log for the first failing job, if we download it completely, doesn't say it is using ninja to build, which means that the CI ran and failed before the change in pytorch/pytorch#32495 was committed.

EDIT2: Here is the test PR where we build without ninja. This one also fails in the same way as reported in this issue, so ninja is probably unrelated.

[seemethere]

Do we have any other leads on why this is happening? Perhaps adding some debugging information to the pytest run would provide some insight into what test it's actually timing out on

[seemethere]

After submitting #1880 I now believe the culprit we should be looking at is something that runs after test/test_quantized_models.py::ModelTester::test_googlenet.

From these logs:

test/test_ops.py::DeformConvTester::test_forward_cpu_contiguous PASSED   [ 67%]
test/test_ops.py::DeformConvTester::test_forward_cpu_non_contiguous PASSED [ 68%]
test/test_ops.py::DeformConvTester::test_forward_cuda_contiguous SKIPPED [ 68%]
test/test_ops.py::DeformConvTester::test_forward_cuda_non_contiguous SKIPPED [ 68%]
test/test_quantized_models.py::ModelTester::test_googlenet PASSED        [ 69%]

Too long with no output (exceeded 10m0s): context deadline exceeded

[zou3519]

Is this reproducible locally?

[seemethere]

I think it’s highly dependent on performance. If the test takes longer than 10 min to run then it’ll time out like this. Were there any tests within that file that were changed recently?

[fmassa]

test_ops.py does call C++ extensions, but as you mentioned it might not be test_ops the culprit.

If the issue is in test_quantized_models, I'd recommend checking with @raghuramank100 to see if anything has changed on the quantization side

[seemethere]

From my local testing the test that takes the longest is Test Inception V3 from the test_quantized_models.py file, which takes about 846s on a devserver

Got this result from running on my branch that outputs junit logs

CU_VERSION=cpu PYTHON_VERSION=3.8 packaging/build_conda.sh

Results are here: results.zip

This would lead me to believe that this test is the culprit.

I've submitted #1885 to skip the test for now and get the nightly build pipeline back on track.

[seemethere]

@fmassa @zou3519 Who would be the person to assign this issue to so we can re-add this test back to the nightly matrix?

Assigning @fmassa in the interim

[fmassa]

Assigning this to @raghuramank100 , who leads the quantization efforts

[datumbox]

This should be fixed by #3196.