Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test some flaky detection models on float64 instead of float32 #7204

Merged
merged 4 commits into from
Feb 9, 2023
Merged

Test some flaky detection models on float64 instead of float32 #7204

merged 4 commits into from
Feb 9, 2023

Conversation

NicolasHug
Copy link
Member

@NicolasHug NicolasHug commented Feb 9, 2023 •
edited by pytorch-bot bot
Loading

Subset of #7114 to push towards a green[er] CI

Still not ideal at all, but it looks like this makes the detection tests pass again.

cc @pmeier

@NicolasHug NicolasHug marked this pull request as ready for review February 9, 2023 11:29
@NicolasHug NicolasHug requested a review from pmeier February 9, 2023 11:29
@NicolasHug
Copy link
Member Author

The last 2 remaining failures are classif-related:

FAILED test/test_models.py::test_classification_model[cuda-resnet101] - AssertionError: Tensor-likes are not close!

Mismatched elements: 1 / 50 (2.0%)
Greatest absolute difference: 3.332275390625 at index (0, 22) (up to 0.1 allowed)
Greatest relative difference: 0.17568308439720962 at index (0, 22) (up to 0.1 allowed)
FAILED test/test_models.py::test_classification_model[cuda-vit_h_14] - RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

pmeier
pmeier approved these changes Feb 9, 2023
View reviewed changes
Copy link
Collaborator

@pmeier pmeier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Nicolas, LGTM! One non-blocking comment below.

test/test_models.py Outdated
model.eval().to(device=dev)
x = _get_image(input_shape=input_shape, real_image=real_image, device=dev)
model.eval().to(device=dev, dtype=dtype)
x = _get_image(input_shape=input_shape, real_image=real_image, device=dev).to(dtype)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we put the dtype parameter into _get_image to avoid an additional copy? This is present in #7130.

@NicolasHug NicolasHug merged commit 8170055 into pytorch:main Feb 9, 2023
@github-actions
Copy link

github-actions bot commented Feb 9, 2023

Hey @NicolasHug!

You merged this PR, but no labels were added. The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py

facebook-github-bot pushed a commit that referenced this pull request Mar 28, 2023
@NicolasHug @facebook-github-bot
[fbsync] Test some flaky detection models on float64 instead of float…
f3cfc49 
…32 (#7204)

Reviewed By: vmoens

Differential Revision: D44416251

fbshipit-source-id: e80c2c1ebde2bad25ab5b6f29e2ac2278e0f4b90
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pmeier pmeier pmeier approved these changes

Assignees
No one assigned
Labels
cla signed module: models module: tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@NicolasHug @pmeier @facebook-github-bot