[CI] gpu test with pytorch nightly

[ydcjeff]

What does this PR do?

Fixes #2090
Creates .drone.jsonnet for multiple testing

DO NOT MERGE THIS UNTIL WE SCALE UP DRONE

[Borda]

let's write it as jsonnet script because just one parameter changes, right?

• https://discourse.drone.io/t/porting-matrix-builds-to-1-0-multi-machine-pipelines/2966
• https://github.com/suzuki-shunsuke/drone-jsonnet-generator

[ydcjeff]

let's write it as jsonnet script because just one parameter changes, right?

* https://discourse.drone.io/t/porting-matrix-builds-to-1-0-multi-machine-pipelines/2966

* https://github.com/suzuki-shunsuke/drone-jsonnet-generator

Yes, only docker image changes.
Okay, will take a look

[Borda]

I think that then we would need to coordinate, because when you add it, ping me and would need to a temporary change the Drone config file, so lets keep the old one and add this jsonnet as a new file...

[Borda]

@ydcjeff we can move forward...

[ydcjeff]

@Borda btw, are we going to run them one after another or in parallel?

[Borda]

well mi mind wait with this one as we are reaching some computational limits on Drone, sometimes. now we have a queue for about 2hours and with running this one it will be even double... so let's prepare it but wait with merging it...

[ydcjeff]

Yea, I am fine with anyway. I have now setup drone to run one after another in jsonnet.
Here's jsonnet to yaml output.

---
kind: pipeline
type: docker
name: torch-GPU

platform:
  os: linux
  arch: amd64

steps:
- name: testing
  image: pytorchlightning/pytorch_lightning:base-cuda-py3.7-torch1.6
  commands:
  - export PATH=$PATH:/root/.local/bin
  - python --version
  - pip install pip -U
  - pip --version
  - nvidia-smi
  - apt-get update && apt-get install -y cmake
  - pip install -r ./requirements/base.txt -q --upgrade-strategy only-if-needed
  - pip install -r ./requirements/devel.txt -q --upgrade-strategy only-if-needed
  - pip install -r ./requirements/examples.txt -q --upgrade-strategy only-if-needed
  - pip list
  - python -c 'import torch ; print(' & '.join([torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())]) if torch.cuda.is_available() else 'only CPU')'
  - coverage run --source pytorch_lightning -m py.test pytorch_lightning tests -v --durations=25
  - python -m py.test benchmarks pl_examples -v --maxfail=2 --durations=0
  - coverage report
  - codecov --token $CODECOV_TOKEN --flags=gpu,pytest --name='GPU-coverage' --env=linux --build $DRONE_BUILD_NUMBER --commit $DRONE_COMMIT
  - python tests/collect_env_details.py
  environment:
    CODECOV_TOKEN:
      from_secret: codecov_token
    HOROVOD_GPU_OPERATIONS: NCCL
    HOROVOD_WITHOUT_MPI: 1
    HOROVOD_WITHOUT_MXNET: 1
    HOROVOD_WITHOUT_TENSORFLOW: 1
    HOROVOD_WITH_GLOO: 1
    HOROVOD_WITH_PYTORCH: 1
    MKL_THREADING_LAYER: GNU
    SLURM_LOCALID: 0

trigger:
  branch:
  - master
  event:
  - push
  - pull_request

---
kind: pipeline
type: docker
name: torch-GPU-nightly

platform:
  os: linux
  arch: amd64

steps:
- name: testing
  image: pytorchlightning/pytorch_lightning:base-cuda-py3.7-torch1.7
  commands:
  - export PATH=$PATH:/root/.local/bin
  - python --version
  - pip install pip -U
  - pip --version
  - nvidia-smi
  - apt-get update && apt-get install -y cmake
  - pip install -r ./requirements/base.txt -q --upgrade-strategy only-if-needed
  - pip install -r ./requirements/devel.txt -q --upgrade-strategy only-if-needed
  - pip install -r ./requirements/examples.txt -q --upgrade-strategy only-if-needed
  - pip list
  - python -c 'import torch ; print(' & '.join([torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())]) if torch.cuda.is_available() else 'only CPU')'
  - coverage run --source pytorch_lightning -m py.test pytorch_lightning tests -v --durations=25
  - python -m py.test benchmarks pl_examples -v --maxfail=2 --durations=0
  - coverage report
  - codecov --token $CODECOV_TOKEN --flags=gpu,pytest --name='GPU-coverage' --env=linux --build $DRONE_BUILD_NUMBER --commit $DRONE_COMMIT
  - python tests/collect_env_details.py
  environment:
    CODECOV_TOKEN:
      from_secret: codecov_token
    HOROVOD_GPU_OPERATIONS: NCCL
    HOROVOD_WITHOUT_MPI: 1
    HOROVOD_WITHOUT_MXNET: 1
    HOROVOD_WITHOUT_TENSORFLOW: 1
    HOROVOD_WITH_GLOO: 1
    HOROVOD_WITH_PYTORCH: 1
    MKL_THREADING_LAYER: GNU
    SLURM_LOCALID: 0

trigger:
  branch:
  - master
  event:
  - push
  - pull_request

depends_on:
- torch-GPU

...

[ydcjeff]

well mi mind wait with this one as we are reaching some computational limits on Drone, sometimes. now we have a queue for about 2hours and with running this one it will be even double... so let's prepare it but wait with merging it...

@Borda regarding with testing nightly, would it be better to start support PT nightly version ahead of one month before stable launch?
also draft those nightly PRs and test them without merging so we do not need to do the follow PRs for fixing. I thought this could save some GPU resources.

For now, there were already 2 PRs for PyTorch nightly, so started thinking about that...

What do you think?

[codecov]

Codecov Report

Merging #3543 (5898c64) into master (6831ba9) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #3543   +/-   ##
======================================
  Coverage      93%     93%           
======================================
  Files         118     118           
  Lines        9018    9018           
======================================
  Hits         8389    8389           
  Misses        629     629           

[edenlightning]

hey @ydcjeff hows it going? any way we can help?

[Borda]

@edenlightning we can merge it as it now, but to activate it (switch source in Drone config) we need to get scalable Drone testing running because this basically double the number of performed tests and already now we are full...

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you need further help see our docs: https://pytorch-lightning.readthedocs.io/en/latest/CONTRIBUTING.html#pull-request or ask the assistance of a core contributor here or on Slack. Thank you for your contributions.

[Borda]

update after #3658

[Borda]

This is not tested, but still, I would merge it as using it as a baseline and debug it later...
For testing, we need a temporary switch Drone setting and see if it works...

[ydcjeff]

Okay 👌

[ydcjeff]

I messed up with my forked repo's master and re-forked again.
Let me know if you want to merge, I can create a new one.

[Borda]

@ydcjeff seems this is not valid as you lost the original fork, mind create it again and refer to this already approved PR 🐰