Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tests for prototype <-> legacy transforms consistency #6514

Merged
merged 2 commits into from
Aug 30, 2022

Conversation

pmeier
Copy link
Collaborator

@pmeier pmeier commented Aug 30, 2022

This PR adds the framework for how we can simply do consistency tests between torchvision.transforms and torchvision.prototype.transforms. One only needs to specify both transforms as well the construction arguments and we get automatic tests for:

  • The prototype transform can be instantiated with the same arguments as its legacy counterpart
  • The prototype transform can transform tensor, feature (features.Image) and PIL images
  • The output of the prototype transform for tensor and PIL images is equal to the output of its legacy counterpart
  • The output of the prototype transform for feature images is equal (minus the type) to its own output for tensor images

This approach is limited to deterministic transforms only. If the transformation includes any kind of randomness, we'll have to add consistency checks manually, e.g. #6511.

To showcase the test framework, I've added all deterministic transformations from the classification references for now. There already was a bug in Resize() that I fixed. I propose we don't add more examples in this PR, but rather merge this fast and add more examples in follow-up PRs.

Copy link
Collaborator

@vfdev-5 vfdev-5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pmeier, lgtm !

@pmeier
Copy link
Collaborator Author

pmeier commented Aug 30, 2022

CI failures are unrelated:

FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_traversable[imagenet-val-True]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_traversable[cub200-train-2011-True]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_traversable[cub200-test-2011-True]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[imagenet-val-0]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[imagenet-val-1]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[cub200-train-2011-0]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[cub200-train-2011-1]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[cub200-test-2011-0]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_data_loader[cub200-test-2011-1]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShufflerIterDataPipe-imagenet-val]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShufflerIterDataPipe-cub200-train-2011]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShufflerIterDataPipe-cub200-test-2011]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShardingFilterIterDataPipe-imagenet-val]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShardingFilterIterDataPipe-cub200-train-2011]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_has_annotations[ShardingFilterIterDataPipe-cub200-test-2011]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_infinite_buffer_size[imagenet-val]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_infinite_buffer_size[cub200-train-2011]
FAILED test/test_prototype_builtin_datasets.py::TestCommon::test_infinite_buffer_size[cub200-test-2011]

Given that the failures are only happening for imagenet and cub200, my gut says this has something to do with #6076. cc @NicolasHug @ejguan @NivekT

@pmeier pmeier merged commit 0d69e35 into pytorch:main Aug 30, 2022
@pmeier pmeier deleted the proto-transform-consistency branch August 30, 2022 09:57
facebook-github-bot pushed a commit that referenced this pull request Aug 30, 2022
…6514)

Summary:
* add consistency checks for prototype and legacy transforms

* fix Resize

Reviewed By: NicolasHug

Differential Revision: D39131006

fbshipit-source-id: 14578e3f91916e5873d377cffb87c9b716e89773
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants