Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix datamodules ImportError and add tests #380

Merged

Conversation

akihironitta
Copy link
Contributor

What does this PR do?

Fixes #379.

Bacause raising ModuleNotFoundError in pl_bolts/datamodules/sklearn_datamodule.py makes from pl_bolts.datamodules import SklearnDataModule raise ImportError as reported in the issue, this PR fixes the problem by replacing it with warn_missing_pkg.

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?
  • Did you verify new and existing tests pass locally with your changes?
  • If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

  • Is this pull request ready for review? (if not, please submit in draft mode)

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@akihironitta akihironitta added fix fixing issues... datamodule Anything related to datamodules labels Nov 19, 2020
@codecov
Copy link

codecov bot commented Nov 19, 2020

Codecov Report

Merging #380 (37cd8d3) into master (70833f0) will decrease coverage by 0.93%.
The diff coverage is 93.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #380      +/-   ##
==========================================
- Coverage   82.03%   81.10%   -0.94%     
==========================================
  Files         100      100              
  Lines        5639     5725      +86     
==========================================
+ Hits         4626     4643      +17     
- Misses       1013     1082      +69     
Flag Coverage Δ
cpu 25.22% <81.25%> (+0.67%) ⬆️
pytest 25.22% <81.25%> (+0.67%) ⬆️
unittests 80.47% <81.25%> (-0.82%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pl_bolts/datasets/kitti_dataset.py 35.29% <83.33%> (+4.85%) ⬆️
pl_bolts/datamodules/experience_source.py 95.86% <100.00%> (+0.14%) ⬆️
pl_bolts/datasets/imagenet_dataset.py 19.41% <100.00%> (+0.95%) ⬆️
..._bolts/models/self_supervised/simclr/transforms.py 80.88% <0.00%> (-19.12%) ⬇️
pl_bolts/datamodules/__init__.py 77.35% <0.00%> (-11.33%) ⬇️
...lts/models/self_supervised/simclr/simclr_module.py 71.49% <0.00%> (-11.32%) ⬇️
pl_bolts/losses/self_supervised_learning.py 71.33% <0.00%> (-6.37%) ⬇️
pl_bolts/datasets/base_dataset.py 95.45% <0.00%> (-4.55%) ⬇️
...l_bolts/models/self_supervised/byol/byol_module.py 83.90% <0.00%> (-1.04%) ⬇️
pl_bolts/models/self_supervised/resnets.py 92.04% <0.00%> (-0.74%) ⬇️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70833f0...37cd8d3. Read the comment docs.

@Borda
Copy link
Member

Borda commented Nov 19, 2020

shall we add some tests that we do not break these imports in the future?

@akihironitta
Copy link
Contributor Author

@Borda Yes, let's.

@Borda
Copy link
Member

Borda commented Nov 19, 2020

@Borda Yes, let's.

may we add it as part of this PR?
probably for this test, we would need to mock some packages aka they are not installed...
there shall be a way wrap such test and inside call these testing imports

@akihironitta
Copy link
Contributor Author

@Borda Just added the import tests when optional packages are unavailable. How does it look?
https://github.com/PyTorchLightning/pytorch-lightning-bolts/blob/ff8cd589328a32557c9f1f68356f94a296f07ab4/tests/datamodules/test_imports.py

@akihironitta
Copy link
Contributor Author

We still have similar problems in importing datamodules when dependencies are not met... e.g. from pl_bolts.datamodules import DiscountedExperienceSource still fails without gym installed, so let me extend this PR to fix the problem for all datamodules.

@akihironitta akihironitta changed the title Fix SklearnDataModule ImportError Fix datamodules ImportError Nov 20, 2020
@akihironitta akihironitta changed the title Fix datamodules ImportError Fix datamodules ImportError [wip] Nov 20, 2020
@Borda
Copy link
Member

Borda commented Nov 20, 2020

@Borda Just added the import tests when optional packages are unavailable. How does it look?
https://github.com/PyTorchLightning/pytorch-lightning-bolts/blob/ff8cd589328a32557c9f1f68356f94a296f07ab4/tests/datamodules/test_imports.py

I may be thinking about some automatic way lets say parse the __init__ but that would require more work of finding dependency in the follow-up file... co this is fine :]
cc: @oke-aditya

@oke-aditya
Copy link
Contributor

oke-aditya commented Nov 20, 2020

I too will give a thought. Solution doesn't seem very straightforward to me 😅

@akihironitta akihironitta changed the title Fix datamodules ImportError [wip] Fix datamodules ImportError Nov 20, 2020
@akihironitta
Copy link
Contributor Author

I'm aware that there is inconsistency in how we check if optional packages are available (i.e. try: ... except ModuleNotFoundError: ... and importlib.util.find_spec("some_package")). I'm trying to refactor datamodules in #338. Also, in #346, I'm suggesting to implement a context manager class which will reduce repetitive import checks all over the codebase, so for this PR, let's only make datamodules importing work.

@akihironitta akihironitta requested a review from Borda November 20, 2020 12:58
@akihironitta akihironitta changed the title Fix datamodules ImportError Fix datamodules ImportError and add tests Nov 20, 2020
pl_bolts/datamodules/kitti_datamodule.py Show resolved Hide resolved
pl_bolts/datamodules/sklearn_datamodule.py Show resolved Hide resolved
pl_bolts/datamodules/sklearn_datamodule.py Outdated Show resolved Hide resolved
tests/datamodules/test_imports.py Show resolved Hide resolved
Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
@akihironitta
Copy link
Contributor Author

@Borda mind having a look again?

@Borda Borda merged commit ecbb82a into Lightning-Universe:master Nov 24, 2020
@akihironitta akihironitta deleted the fix/raise-error-when-unavailable branch January 4, 2021 15:37
@Borda Borda added this to the v0.3 milestone Jan 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datamodule Anything related to datamodules fix fixing issues...
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cannot import name 'SklearnDataModule' from 'pl_bolts.datamodules'
4 participants