Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Azure - multiple configs #12984

Merged
merged 25 commits into from
May 14, 2022
Merged

CI: Azure - multiple configs #12984

merged 25 commits into from
May 14, 2022

Conversation

Borda
Copy link
Member

@Borda Borda commented May 5, 2022

What does this PR do?

Followup #12373
Extend testing configs for GPU also for the latest version
cc: @akihironitta

Closes #12955.

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

cc @carmocca @akihironitta @Borda

@Borda Borda added the ci Continuous Integration label May 5, 2022
@Borda Borda requested review from tchaton and carmocca as code owners May 5, 2022 15:26
@Borda Borda requested review from akihironitta, kaushikb11, otaj and a team May 5, 2022 15:27
@Borda
Copy link
Member Author

Borda commented May 5, 2022

it is expected to fail the PT1.11 until docker is unpin in #12373

@Borda Borda mentioned this pull request May 5, 2022
12 tasks
@mergify mergify bot added the has conflicts label May 6, 2022
@mergify mergify bot removed the has conflicts label May 6, 2022
Copy link
Contributor

@akihironitta akihironitta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

.azure-pipelines/gpu-tests.yml Outdated Show resolved Hide resolved
Co-authored-by: Akihiro Nitta <nitta@akihironitta.com>
@Borda Borda requested a review from akihironitta May 6, 2022 04:41
@Borda Borda enabled auto-merge (squash) May 6, 2022 04:41
@Borda Borda requested review from awaelchli and a team May 6, 2022 09:55
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
@akihironitta akihironitta added this to the 1.6.x milestone May 7, 2022
@mergify mergify bot removed the has conflicts label May 10, 2022
@akihironitta
Copy link
Contributor

akihironitta commented May 10, 2022

For the note, the current failures are:

6685:2022-05-10T00:49:27.6990597Z FAILED tests/strategies/test_ddp_fully_sharded_native.py::test_fully_sharded_native_strategy_sync_batchnorm
6797:2022-05-10T00:49:38.9204765Z FAILED tests/strategies/test_ddp_fully_sharded_native.py::test_fully_sharded_native_strategy_checkpoint
6911:2022-05-10T00:49:50.1150632Z FAILED tests/strategies/test_ddp_fully_sharded_native.py::test_fully_sharded_native_strategy_checkpoint_multi_gpus
10517:2022-05-10T00:59:43.0355034Z FAILED tests/strategies/test_deepspeed_strategy.py::test_deepspeed_with_meta_device
11307:2022-05-10T00:59:43.5558192Z FAILED tests/strategies/test_deepspeed_strategy.py::test_deepspeed_with_meta_device

These native FSDP failures should be fixed in #12985, so in this PR, we're going to just disable them.

Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
@akihironitta
Copy link
Contributor

This PR should finally be ready for merge after #12870 lands.

tests/standalone_tests.sh Outdated Show resolved Hide resolved
@mergify mergify bot added the ready PRs ready to be merged label May 13, 2022
@akihironitta
Copy link
Contributor

The blocker #12870 has been resolved, so this PR should be ready for getting merged.

@Borda @carmocca Could you have another look if needed?

@Borda Borda enabled auto-merge (squash) May 14, 2022 01:44
@Borda Borda merged commit fab2ff3 into master May 14, 2022
@Borda Borda deleted the ci/gpus branch May 14, 2022 01:59
Comment on lines +23 to +25
'PyTorch - LTS':
image: "pytorchlightning/pytorch_lightning:base-cuda-py3.7-torch1.8"
'PyTorch - stable':
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we shorten this text?

Suggested change
'PyTorch - LTS':
image: "pytorchlightning/pytorch_lightning:base-cuda-py3.7-torch1.8"
'PyTorch - stable':
'LTS':
image: "pytorchlightning/pytorch_lightning:base-cuda-py3.7-torch1.8"
'stable':

carmocca added a commit that referenced this pull request May 24, 2022
* CI: Azure - multiple configs
* names
* benchmark
* Apply suggestions from code review

Co-authored-by: Akihiro Nitta <nitta@akihironitta.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
carmocca added a commit that referenced this pull request May 31, 2022
* CI: Azure - multiple configs
* names
* benchmark
* Apply suggestions from code review

Co-authored-by: Akihiro Nitta <nitta@akihironitta.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
lexierule pushed a commit that referenced this pull request Jun 1, 2022
* CI: Azure - multiple configs
* names
* benchmark
* Apply suggestions from code review

Co-authored-by: Akihiro Nitta <nitta@akihironitta.com>
Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
@rohitgr7 rohitgr7 mentioned this pull request Jul 1, 2022
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Continuous Integration ready PRs ready to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add testing with PyTorch 1.11 on GPUs in CI
5 participants