Remove CUDA 11.7 builds; add 11.8

[ptrblck]

CC @atalman @malfet

cc @seemethere

This PR will be removing cuda 11.7 builds from matrix: pytorch/test-infra#4205

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7616

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 32 New Failures

As of commit d7e420d:

NEW FAILURES - The following jobs have failed:

• build / linux-job (gh)
• c-source / linux-job (gh)
• linux (linux.12xlarge, cpu) / linux-job (gh)
• linux (linux.g5.4xlarge.nvidia.gpu, cuda, 11.8) / linux-job (gh)
• macos (macos-12) / macos-job (gh)
• macos (macos-m1-12) / macos-job (gh)
• onnx / linux-job (gh)
• python-source-and-configs / linux-job (gh)
• python-types / linux-job (gh)
• unittests-extended / linux-job (gh)
• unittests-linux (3.10, linux.12xlarge, cpu) / linux-job (gh)
• unittests-linux (3.11, linux.12xlarge, cpu) / linux-job (gh)
• unittests-linux (3.8, linux.12xlarge, cpu) / linux-job (gh)
• unittests-linux (3.8, linux.g5.4xlarge.nvidia.gpu, cuda, 11.8) / linux-job (gh)
• unittests-linux (3.9, linux.12xlarge, cpu) / linux-job (gh)
• unittests-macos (3.10, macos-12) / macos-job (gh)
• unittests-macos (3.11, macos-12) / macos-job (gh)
• unittests-macos (3.8, macos-12) / macos-job (gh)
• unittests-macos (3.8, macos-m1-12) / macos-job (gh)
• unittests-macos (3.9, macos-12) / macos-job (gh)
• unittests-prototype (3.10, linux.12xlarge, cpu) / linux-job (gh)
• unittests-prototype (3.11, linux.12xlarge, cpu) / linux-job (gh)
• unittests-prototype (3.8, linux.12xlarge, cpu) / linux-job (gh)
• unittests-prototype (3.8, linux.g5.4xlarge.nvidia.gpu, cuda, 11.8) / linux-job (gh)
• unittests-prototype (3.9, linux.12xlarge, cpu) / linux-job (gh)
• unittests-windows (3.10, windows.4xlarge, cpu) / windows-job (gh)
• unittests-windows (3.11, windows.4xlarge, cpu) / windows-job (gh)
• unittests-windows (3.8, windows.4xlarge, cpu) / windows-job (gh)
• unittests-windows (3.8, windows.g5.4xlarge.nvidia.gpu, cuda, 11.8) / windows-job (gh)
• unittests-windows (3.9, windows.4xlarge, cpu) / windows-job (gh)
• windows (windows.4xlarge, cpu) / windows-job (gh)
• windows (windows.g5.4xlarge.nvidia.gpu, cuda, 11.8) / windows-job (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[pmeier]

Hey @ptrblck, we are in the process of killing CircleCI in favor of GitHub Actions (#7611). Thus, your patch would be removed soon. Plus, it is missing the parts on GitHub Actions that also need to be patched. I'll send a few new commits here for future reference.

[pmeier]

In the future, we also need to set this on the CMake workflow as I did in 79ae3e6 / #7417.

[pmeier]

Linux GPU failure looks valid

Traceback (most recent call last):
  File "/work/test/test_models.py", line 705, in test_classification_model
    _assert_expected(out.cpu(), model_name, prec=prec)
  File "/work/test/test_models.py", line 155, in _assert_expected
    torch.testing.assert_close(output, expected, rtol=rtol, atol=atol, check_dtype=False, check_device=False)
  File "/opt/conda/envs/ci/lib/python3.8/site-packages/torch/testing/_comparison.py", line 1511, in assert_close
    raise error_metas[0].to_error(msg)
AssertionError: Tensor-likes are not close!

Mismatched elements: 1 / 50 (2.0%)
Greatest absolute difference: 5.10198974609375 at index (0, 22) (up to 0.2 allowed)
Greatest relative difference: 0.2689853608608246 at index (0, 22) (up to 0.2 allowed)

Since the CPU test is just fine, this means that our GPU output now diverges from the CUDA 11.7 one.

We already have quite a few hacks in

vision/test/test_models.py

Line 676 in bd4471c

def test_classification_model(model_fn, dev):

@NicolasHug do you know if we had something like above before?

[atalman]

Linux GPU failure looks valid

Traceback (most recent call last):
  File "/work/test/test_models.py", line 705, in test_classification_model
    _assert_expected(out.cpu(), model_name, prec=prec)
  File "/work/test/test_models.py", line 155, in _assert_expected
    torch.testing.assert_close(output, expected, rtol=rtol, atol=atol, check_dtype=False, check_device=False)
  File "/opt/conda/envs/ci/lib/python3.8/site-packages/torch/testing/_comparison.py", line 1511, in assert_close
    raise error_metas[0].to_error(msg)
AssertionError: Tensor-likes are not close!

Mismatched elements: 1 / 50 (2.0%)
Greatest absolute difference: 5.10198974609375 at index (0, 22) (up to 0.2 allowed)
Greatest relative difference: 0.2689853608608246 at index (0, 22) (up to 0.2 allowed)

Since the CPU test is just fine, this means that our GPU output now diverges from the CUDA 11.7 one.

We already have quite a few hacks in

vision/test/test_models.py

Line 676 in bd4471c

def test_classification_model(model_fn, dev):

@NicolasHug do you know if we had something like above before?

@NicolasHug @pmeier should we create a separate issue for this and unblock this PR ?

[NicolasHug]

I'm not sure, the failure is rather concerning especially considering it happens on such a simple arch like ResNet. Are we sure this isn't an upstream regression?

[github-actions]

Hey @atalman!

You merged this PR, but no labels were added. The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py