Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reported accuracy mismatch for vgg11_bn #313

Open
nps1ngh opened this issue Dec 21, 2022 · 2 comments · May be fixed by #319
Open

Reported accuracy mismatch for vgg11_bn #313

nps1ngh opened this issue Dec 21, 2022 · 2 comments · May be fixed by #319

Comments

@nps1ngh
Copy link

nps1ngh commented Dec 21, 2022

The hub documentation here: https://pytorch.org/hub/pytorch_vision_vgg/
reports the vgg11_bn top-1 error to be 26.70. That is, a top-1 accuracy of 73.30.
This is almost as good as the resnet34 reported here: https://pytorch.org/vision/main/models/generated/torchvision.models.resnet34.html

But the main problem is that torchvision reports a top-1 accuracy of 70.37 for the it here: https://pytorch.org/vision/main/models/generated/torchvision.models.vgg11_bn.html

I guess the latter reported accuracy is correct (70.37) and the one reported here by hub is incorrect?

Or am I missing something? Is this possibly related to pytorch/vision#223? (It's a really old issue though.)

Thanks!

@NicolasHug
Copy link
Member

NicolasHug commented Jan 9, 2023

Thanks for the report @nps1ngh . Good catch!

For all torchivsion models, the correct reference for accuracies should be this table: https://pytorch.org/vision/main/models.html#table-of-all-available-classification-weights (or the specific model pages like the one you gave above, which has the same info). Unfortunately, the ones reported on the torchhub website may go out of date. I opened #318 to try to think of solutions / mitigations.

Meanwhile, it's probably worth updating the VGG table with the latest correct values. LMK if you would you like to open a PR for that. Thank you!

@nps1ngh
Copy link
Author

nps1ngh commented Jan 9, 2023

Sure, I'll open one!

@nps1ngh nps1ngh linked a pull request Jan 9, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants