Skip to content

Conversation

@bhimrazy
Copy link
Collaborator

@bhimrazy bhimrazy commented Oct 15, 2025

What does this PR do?

This PR has the following changes:

  • Reduced GPU test time by approximately 25–30 minutes.
    Two of these tests were taking more than 20 minutes but, when run manually, finished within 5 seconds.

    • test_tensorrt_save_ir_type[ts-ScriptModule]
    • test_tensorrt_export_reload[ts-torchscript]
      image

    The delay in those tests was mainly caused by CUDA_LAUNCH_BLOCKING=1.

    Before:
    image
    After:
    image

  • Fixes a typo by replacing $(PACKAGE_NAME) with ${PACKAGE_NAME}.

  • Fix: unzip not found — used in ci

    .actions/pull_legacy_checkpoints.sh: line 10: unzip: command not found
    total 0
    + cd tests/legacy
    + ls -lh checkpoints/
    total 0
    

    unzip -o tests/legacy/checkpoints.zip -d tests/legacy/

Note: It seems there are still ci/litjob failures caused by port issues, which require a rerun to pass.

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist
  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

@bhimrazy bhimrazy changed the title chore debug gpu tests [wip]: chore/debug gpu tests Oct 15, 2025
@SkafteNicki SkafteNicki added the ci Continuous Integration label Oct 20, 2025
@bhimrazy bhimrazy marked this pull request as draft October 20, 2025 07:27
@bhimrazy
Copy link
Collaborator Author

bhimrazy commented Nov 2, 2025

It seems the delay in those tests is caused by CUDA_LAUNCH_BLOCKING=1.

Test Code

export CUDA_LAUNCH_BLOCKING=0
time python -c "
import torch
from lightning.pytorch.demos.boring_classes import BoringModel
model = BoringModel()
model.example_input_array = torch.randn((4, 32))
model.cuda()
print('Starting compilation with CI env vars...')
import time
start = time.time()
ret = model.to_tensorrt(ir='ts')
end = time.time()
print(f'Compilation took {end - start:.2f} seconds')
"

Almost 20 mins saved

@bhimrazy bhimrazy mentioned this pull request Nov 3, 2025
7 tasks
@bhimrazy bhimrazy changed the title [wip]: chore/debug gpu tests chore(ci/litjob): fix and optimize GPU related tests Nov 5, 2025
@bhimrazy bhimrazy changed the title chore(ci/litjob): fix and optimize GPU related tests chore(ci/litjob): fix and optimize GPU related tests Nov 5, 2025
@bhimrazy bhimrazy marked this pull request as ready for review November 5, 2025 03:14
@Borda Borda merged commit 0bd1993 into Lightning-AI:master Nov 5, 2025
18 checks passed
@bhimrazy bhimrazy deleted the debug-gpu-tests branch November 5, 2025 08:59
justusschock pushed a commit that referenced this pull request Nov 5, 2025
* fix: add unzip to dependencies in pytorch workflow
* fix: update variable interpolation for PACKAGE_NAME in YAML workflows
* fix: update variable interpolation for PACKAGE_NAME in fabric workflow
* fix: disable CUDA_LAUNCH_BLOCKING to speed up tests
* apply suggestion
justusschock pushed a commit that referenced this pull request Nov 5, 2025
* fix: add unzip to dependencies in pytorch workflow
* fix: update variable interpolation for PACKAGE_NAME in YAML workflows
* fix: update variable interpolation for PACKAGE_NAME in fabric workflow
* fix: disable CUDA_LAUNCH_BLOCKING to speed up tests
* apply suggestion
justusschock added a commit that referenced this pull request Nov 5, 2025
* Remove Support For Deprecated Habana (#21327)

* remove habana

* changelog

* remove from install

* update docs

* update

* update

* pytest match

* Add name to accelerator interface (#21325)

* changelog + rebase

* version.info bumo

* fix: Checkout and Git LFS fetch issue for pytorch docs workflow (#21219)

* ci: `uv` for `docs-build` (#21206)

replace pip with uv

Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>

* fix: failing ci linkcheck in markdown (#21222)

Update timeout and retry configuration
Increased timeout and adjusted retry settings.

* simplify lit CI config (#21220)

* simplify lit CI config

* Empty-Commit

* Empty-Commit

* Update

---------

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* fix:docs build errors (#21286)

* docs: add 'typing.Union' to nitpick_ignore_regex in Sphinx configuration

* docs: add Supermicro link to linkcheck_ignore due to 403 error

* [Hotfix] Remove reference to `torch.jit.export` (#21292)

update docs

* Docs: drop deprecated project (#21250)

* drop dumping wheels after UV switch (#21217)

* chore(`ci/litjob`): fix and optimize GPU related tests (#21290)

* fix: add unzip to dependencies in pytorch workflow
* fix: update variable interpolation for PACKAGE_NAME in YAML workflows
* fix: update variable interpolation for PACKAGE_NAME in fabric workflow
* fix: disable CUDA_LAUNCH_BLOCKING to speed up tests
* apply suggestion

* remove integrations from Dockerfile

* Update readme (#21263)

Update LitServe description for clarity

* Update heading from 'Why PyTorch Lightning?' to 'Convert from PyTorch to PyTorch Lightning?'

* Fix heading formatting in README.md

* Add UTM parameters to links in README

Updated links in README to include UTM parameters for tracking.

* readme

readme

* readme updates

readme updates

* update readme

Update readme

* update

* update

* update

* update

* resolve docker builds

* update

* update

* update

* update

* update

* update

* update

---------

Co-authored-by: Bhimraj Yadav <bhimrajyadav977@gmail.com>
Co-authored-by: Shion Matsumoto <smatsumoto97@gmail.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Deependu <deependujha21@gmail.com>
Co-authored-by: Nicki Skafte Detlefsen <skaftenicki@gmail.com>
Co-authored-by: William Falcon <waf2107@columbia.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Continuous Integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants