Add CUDA 11.3 and fix broken cuTENSOR variations #658

kmaehashi · 2021-05-20T12:34:51Z

No description provided.

kmaehashi · 2021-05-20T12:35:06Z

pfnCI, test this please.

kmaehashi · 2021-05-20T12:54:33Z

pfnCI, test this please.

docker.py

kmaehashi · 2021-05-20T12:55:19Z

pfnCI, test this please.

chainer-ci · 2021-05-20T16:06:11Z

Jenkins CI test (for commit 77a4aee, target branch master) failed with status FAILURE.

kmaehashi · 2021-05-21T03:42:50Z

pfnCI, test this please.

kmaehashi · 2021-05-21T03:54:56Z

pfnCI, test this please.

kmaehashi · 2021-05-21T04:20:43Z

pfnCI, test this please.

chainer-ci · 2021-05-21T06:46:45Z

Jenkins CI test (for commit 29b2984, target branch master) failed with status FAILURE.

kmaehashi · 2021-05-21T07:24:37Z

Blocked by CUDA driver update in Jenkins slave. -> done

kmaehashi · 2021-05-24T15:45:06Z

pfnCI, test this please.

chainer-ci · 2021-05-24T22:09:03Z

Jenkins CI test (for commit 29b2984, target branch master) failed with status FAILURE.

kmaehashi · 2021-05-25T01:58:22Z

pfnCI, test this please.

chainer-ci · 2021-05-25T04:28:34Z

Jenkins CI test (for commit 29b2984, target branch master) failed with status FAILURE.

kmaehashi · 2021-06-02T09:08:49Z

cupy/cupy#5264 got merged!
pfnCI, test this please.

chainer-ci · 2021-06-02T12:43:14Z

Jenkins CI test (for commit 29b2984, target branch master) failed with status FAILURE.

kmaehashi · 2021-06-02T12:44:19Z

pfnCI, test this please.

chainer-ci · 2021-06-02T15:14:08Z

Jenkins CI test (for commit 29b2984, target branch master) failed with status FAILURE.

kmaehashi · 2021-06-02T15:39:26Z

Hmm there are still some async mempool failures...

00:13:32 FAILED cupy_tests/cuda_tests/test_memory.py::TestAllocator_param_1_{mempool='MemoryAsyncPool'}::test_allocator_thread_local
00:13:32 FAILED cupy_tests/cuda_tests/test_memory.py::TestAllocator_param_1_{mempool='MemoryAsyncPool'}::test_set_allocator
00:13:32 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_free_bytes
00:13:32 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_total_bytes
00:13:32 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_total_bytes2
00:13:32 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_total_bytes_stream
00:13:32 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_used_bytes
00:13:32 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_used_bytes2
00:13:32 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_used_bytes_stream

leofang · 2021-06-02T16:27:50Z

I could try to take a look later tonight 😅

leofang · 2021-06-02T21:46:15Z

Hmmm I ran pytest tests/cupy_tests/ on bare metal but couldn't reproduce the errors; all tests in test_memory.py passed. @kmaehashi is the CI hardware resource shared by more than one runners?

kmaehashi · 2021-06-03T04:07:52Z

@leofang Ah yes, several tests may run concurrently on the same device in Jenkins.

kmaehashi · 2021-06-03T13:44:25Z

pfnCI, test this please.

leofang · 2021-06-03T14:13:16Z

Let's see if cupy/cupy#5308 fixes it...🤞

chainer-ci · 2021-06-03T16:00:57Z

Jenkins CI test (for commit 29b2984, target branch master) failed with status FAILURE.

leofang · 2021-06-03T16:10:10Z

Let's see if cupy/cupy#5308 fixes it...🤞

Well, only part of it...

11:51:03 =========================== short test summary info ============================
11:51:03 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_free_bytes
11:51:03 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_total_bytes
11:51:03 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_total_bytes2
11:51:03 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_total_bytes_stream
11:51:03 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_used_bytes
11:51:03 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_used_bytes2
11:51:03 FAILED cupy_tests/cuda_tests/test_memory.py::TestMemoryAsyncPool::test_used_bytes_stream
11:51:03 = 7 failed, 80662 passed, 15834 skipped, 641 deselected, 74 xfailed, 1464 warnings in 5759.79s (1:35:59) =

Let me check later...

leofang · 2021-06-07T04:07:20Z

Should we mark these failing tests xfail for the time being? I do not immediately see what else could go wrong and I'm wondering if it's due to architecture difference.

kmaehashi · 2021-06-17T05:42:27Z

Should we mark these failing tests xfail for the time being? I do not immediately see what else could go wrong and I'm wondering if it's due to architecture difference.

Agree, sent a PR. cupy/cupy#5350

kmaehashi · 2021-06-17T08:32:59Z

pfnCI, test this please.

chainer-ci · 2021-06-17T11:53:39Z

Jenkins CI test (for commit 29b2984, target branch master) failed with status FAILURE.

kmaehashi · 2021-06-17T12:43:19Z

pfnCI, test this please.

chainer-ci · 2021-06-17T15:44:17Z

Jenkins CI test (for commit 29b2984, target branch master) failed with status FAILURE.

kmaehashi · 2021-06-17T15:47:30Z

Jenkins test this please.

chainer-ci · 2021-06-18T02:07:14Z

Jenkins CI test (for commit 29b2984, target branch master) failed with status FAILURE.

kmaehashi · 2021-06-18T02:07:36Z

Test failure is irrelevant.

asi1024 · 2021-06-21T05:06:21Z

LGTM.

add CUDA 11.3

fb9f3e8

kmaehashi mentioned this pull request May 20, 2021

NCCL v2.9.8 #657

Merged

Merge branch 'master' into cuda113

c22f020

kmaehashi changed the title ~~Add CUDA 11.3~~ Add CUDA 11.3 and fix broken cuTENSOR variations May 20, 2021

kmaehashi commented May 20, 2021

View reviewed changes

docker.py Outdated Show resolved Hide resolved

cosmetic fix

77a4aee

kmaehashi mentioned this pull request May 20, 2021

Support CUDA 11.3 cupy/cupy#5095

Closed

21 tasks

add cuda113

57bac94

kmaehashi added 2 commits May 21, 2021 03:53

fix

692b6c1

Merge remote-tracking branch 'origin/cuda113' into cuda113

220abb4

add nccl

29b2984

kmaehashi assigned asi1024 May 21, 2021

kmaehashi mentioned this pull request May 23, 2021

[backport] Add CUDA 11.3 headers cupy/cupy#5232

Merged

kmaehashi mentioned this pull request May 24, 2021

CuPy v10 requires CUDA 10.0 or later #660

Merged

asi1024 approved these changes May 24, 2021

View reviewed changes

kmaehashi mentioned this pull request May 25, 2021

Tentatively remove CUDA 11.3 from support list cupy/cupy#5256

Merged

takagi mentioned this pull request May 26, 2021

NCCL v2.9.9 #661

Merged

leofang mentioned this pull request Jun 2, 2021

Temporarily remove the async pool test from TestAllocator cupy/cupy#5308

Merged

kmaehashi mentioned this pull request Jun 17, 2021

Tentatively skip unstable MemoryPoolAsync tests cupy/cupy#5350

Merged

asi1024 merged commit 459f051 into master Jun 21, 2021

asi1024 deleted the cuda113 branch June 21, 2021 05:06

leofang mentioned this pull request Jun 21, 2021

MemoryAsyncPool test issues (tracker issue) cupy/cupy#5349

Open

Add CUDA 11.3 and fix broken cuTENSOR variations #658

Add CUDA 11.3 and fix broken cuTENSOR variations #658

Conversation

kmaehashi commented May 20, 2021

kmaehashi commented May 20, 2021

kmaehashi commented May 20, 2021

kmaehashi commented May 20, 2021

chainer-ci commented May 20, 2021

kmaehashi commented May 21, 2021

kmaehashi commented May 21, 2021

kmaehashi commented May 21, 2021

chainer-ci commented May 21, 2021

kmaehashi commented May 21, 2021 • edited Loading

kmaehashi commented May 24, 2021

chainer-ci commented May 24, 2021

kmaehashi commented May 25, 2021

chainer-ci commented May 25, 2021

kmaehashi commented Jun 2, 2021

chainer-ci commented Jun 2, 2021

kmaehashi commented Jun 2, 2021

chainer-ci commented Jun 2, 2021

kmaehashi commented Jun 2, 2021

leofang commented Jun 2, 2021

leofang commented Jun 2, 2021

kmaehashi commented Jun 3, 2021

kmaehashi commented Jun 3, 2021

leofang commented Jun 3, 2021

chainer-ci commented Jun 3, 2021

leofang commented Jun 3, 2021

leofang commented Jun 7, 2021

kmaehashi commented Jun 17, 2021

kmaehashi commented Jun 17, 2021

chainer-ci commented Jun 17, 2021

kmaehashi commented Jun 17, 2021

chainer-ci commented Jun 17, 2021

kmaehashi commented Jun 17, 2021

chainer-ci commented Jun 18, 2021

kmaehashi commented Jun 18, 2021

asi1024 commented Jun 21, 2021

kmaehashi commented May 21, 2021 •

edited

Loading