Drone: use nightly build cuda docker images #3658

Borda · 2020-09-25T15:43:50Z

What does this PR do?

Update used docker images to the one we regularly update...
Next PR: switch docker image to PT 1.6 version fro Drone testing

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

After PR merged

remove old images from Docker Hub
https://hub.docker.com/repository/docker/pytorchlightning/pytorch_lightning/tags?page=1&name=cuda-extras

Did you have fun?

Make sure you had fun coding 🙃

Borda · 2020-09-25T16:29:28Z

failing on docker using:


ImportError while loading conftest '/drone/src/tests/conftest.py'.
--
226 | tests/__init__.py:13: in <module>
227 | os.mkdir(TEMP_PATH)
228 | E   PermissionError: [Errno 13] Permission denied: '/drone/src/test_temp'
229 | Traceback (most recent call last):
230 | File "/home/flash/miniconda/envs/lightning/lib/python3.7/site-packages/coverage/cmdline.py", line 740, in do_run
231 | runner.run()
232 | File "/home/flash/miniconda/envs/lightning/lib/python3.7/site-packages/coverage/execfile.py", line 247, in run
233 | exec(code, main_mod.__dict__)
234 | File "/home/flash/miniconda/envs/lightning/lib/python3.7/site-packages/py/test.py", line 4, in <module>
235 | sys.exit(pytest.main())
236 | SystemExit: ExitCode.USAGE_ERROR

Borda · 2020-09-25T23:40:23Z

need to rebuild the docker image in #3661

ydcjeff · 2020-09-26T03:15:52Z

Could this error be the non-root user creation?

Borda · 2020-09-26T22:02:16Z

Could this error be the non-root user creation?

it seems so... I ll tttry to get a version with sudo like we have for XLA...

Borda · 2020-09-29T09:04:09Z

@ydcjeff I have built this one locally and pushed it to Docker hub directly to test that it fine to run the tests...

Borda · 2020-09-29T09:23:12Z

seems we have failing test for PR 1.6

FAILED tests/models/test_gpu.py::test_multi_gpu_model_ddp[variation_fit---max_epochs 1 --gpus 2 --distributed_backend ddp]
FAILED tests/models/test_gpu.py::test_multi_gpu_model_ddp[variation_test---max_epochs 1 --gpus 2 --distributed_backend ddp]

is it something you have seen? @awaelchli @williamFalcon
http://35.192.60.23/PyTorchLightning/pytorch-lightning/10779/1/2

mergify · 2020-09-30T12:37:36Z

This pull request is now in conflict... :(

mergify · 2020-10-02T09:27:56Z

This pull request is now in conflict... :(

mergify · 2020-10-06T21:30:27Z

This pull request is now in conflict... :(

codecov · 2020-10-13T20:45:11Z

Codecov Report

Merging #3658 into master will increase coverage by 0%.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #3658   +/-   ##
======================================
  Coverage      93%     93%           
======================================
  Files         111     111           
  Lines        8068    8046   -22     
======================================
- Hits         7487    7467   -20     
+ Misses        581     579    -2

Borda · 2020-10-14T12:35:37Z

probably the DD just needs to set some env variables...

Borda · 2020-10-14T18:36:57Z

pytorch_lightning/accelerators/ddp_accelerator.py

-            del os.environ['WORLD_SIZE']
+        # TODO: try this skip
+        # if 'WORLD_SIZE' in os.environ:
+        #     del os.environ['WORLD_SIZE']


@awaelchli it seems there is no restore anywhere else... so how is it possible it passes on master now?

any idea why the env deletion was added here? especially after training...

pep8speaks · 2020-10-14T19:58:41Z

Hello @Borda! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-10-26 05:32:38 UTC

Borda · 2020-10-16T22:57:42Z

when I run the tests on similar machine...

=================================================================== short test summary info =====================================================================
FAILED tests/backends/test_ddp.py::test_multi_gpu_model_ddp_fit_only[--max_epochs 1 --gpus 2 --distributed_backend ddp] - FileNotFoundError: [Errno 2] No such ...
FAILED tests/backends/test_ddp.py::test_multi_gpu_model_ddp_test_only[--max_epochs 1 --gpus 2 --distributed_backend ddp] - FileNotFoundError: [Errno 2] No such...
FAILED tests/backends/test_ddp.py::test_multi_gpu_model_ddp_fit_test[--max_epochs 1 --gpus 2 --distributed_backend ddp] - FileNotFoundError: [Errno 2] No such ...
FAILED tests/models/test_amp.py::test_amp_with_apex - RuntimeError: CUDA error: no kernel image is available for execution on the device
FAILED tests/trainer/test_trainer.py::test_gradient_clipping_fp16 - AssertionError: Gradient norm != 1.0: nan
FAILED tests/trainer/optimization/test_manual_optimization.py::test_multiple_optimizers_manual_native_amp - RuntimeError: unscale_() has already been called on...
FAILED tests/trainer/optimization/test_manual_optimization.py::test_multiple_optimizers_manual_apex - RuntimeError: CUDA error: no kernel image is available fo...
======================================== 7 failed, 1351 passed, 19 skipped, 1 xfailed, 2544 warnings in 603.30s (0:10:03) ========================================

tchaton

Great Addition :)

Borda added the ci Continuous Integration label Sep 25, 2020

Borda force-pushed the ci/drone-pt1.6 branch from 0853c34 to 28d51af Compare September 29, 2020 08:52

Borda marked this pull request as ready for review September 29, 2020 09:03

mergify bot requested a review from a team September 29, 2020 09:03

Borda changed the title ~~upgrade PT 1.6 version fro Drone testing~~ [wip] upgrade PT 1.5 version fro Drone testing Sep 29, 2020

Borda force-pushed the ci/drone-pt1.6 branch from 6f4da83 to d146770 Compare September 29, 2020 21:25

Borda changed the title ~~[wip] upgrade PT 1.5 version fro Drone testing~~ [blocked by #3739] upgrade PT 1.6 version fro Drone testing Sep 30, 2020

Borda changed the title ~~[blocked by #3739] upgrade PT 1.6 version fro Drone testing~~ upgrade PT 1.6 version fro Drone testing Oct 1, 2020

Borda force-pushed the ci/drone-pt1.6 branch 2 times, most recently from db49fa9 to e739764 Compare October 1, 2020 21:21

Borda force-pushed the ci/drone-pt1.6 branch from e739764 to 59c1f09 Compare October 2, 2020 19:52

Borda force-pushed the ci/drone-pt1.6 branch 4 times, most recently from e15e717 to 21b635d Compare October 13, 2020 20:43

Borda commented Oct 14, 2020

View reviewed changes

Borda marked this pull request as draft October 14, 2020 20:02

prune

574257f

Borda marked this pull request as ready for review October 25, 2020 18:31

Borda requested review from SeanNaren, ananyahjha93, awaelchli, justusschock, nateraw, tchaton, teddykoker and williamFalcon as code owners October 25, 2020 18:31

Borda added 4 commits October 25, 2020 19:34

CI

ddd9672

CI

f1f6843

CI

a5d049e

CI

0192dbc

awaelchli approved these changes Oct 25, 2020

View reviewed changes

mergify bot requested a review from a team October 25, 2020 19:12

Borda requested review from rohitgr7, s-rog, ydcjeff and yukw777 October 25, 2020 23:19

ydcjeff approved these changes Oct 26, 2020

View reviewed changes

Merge branch 'master' into ci/drone-pt1.6

959bd2a

mergify bot requested a review from a team October 26, 2020 05:32

justusschock approved these changes Oct 26, 2020

View reviewed changes

tchaton approved these changes Oct 26, 2020

View reviewed changes

ydcjeff mentioned this pull request Oct 26, 2020

Fix failing conda CI #4359

Closed

1 task

SeanNaren merged commit ce8abd6 into master Oct 26, 2020

SeanNaren deleted the ci/drone-pt1.6 branch October 26, 2020 10:47

ydcjeff mentioned this pull request Oct 27, 2020

Switch to PyTorch 1.6 in Drone CI #4393

Merged

1 task

Borda mentioned this pull request Nov 22, 2020

speed-up too long tests #2041

Closed

Drone: use nightly build cuda docker images #3658

Drone: use nightly build cuda docker images #3658

Uh oh!

Conversation

Borda commented Sep 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

PR review

After PR merged

Did you have fun?

Uh oh!

Borda commented Sep 25, 2020

Uh oh!

Borda commented Sep 25, 2020

Uh oh!

ydcjeff commented Sep 26, 2020

Uh oh!

Borda commented Sep 26, 2020

Uh oh!

Borda commented Sep 29, 2020

Uh oh!

Borda commented Sep 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented Sep 30, 2020

Uh oh!

mergify bot commented Oct 2, 2020

Uh oh!

mergify bot commented Oct 6, 2020

Uh oh!

codecov bot commented Oct 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Borda commented Oct 14, 2020

Uh oh!

Borda Oct 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Borda Oct 14, 2020

Choose a reason for hiding this comment

Uh oh!

pep8speaks commented Oct 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated at 2020-10-26 05:32:38 UTC

Uh oh!

Borda commented Oct 16, 2020

Uh oh!

tchaton left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Borda commented Sep 25, 2020 •

edited

Loading

Borda commented Sep 29, 2020 •

edited

Loading

codecov bot commented Oct 13, 2020 •

edited

Loading

Borda Oct 14, 2020 •

edited

Loading

pep8speaks commented Oct 14, 2020 •

edited

Loading