Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When I run the Base_OC_Module at a certain iteration, it will stop. #16

Open
yu-changqian opened this issue Sep 27, 2018 · 3 comments
Open

Comments

@yu-changqian
Copy link

yu-changqian commented Sep 27, 2018

Pytorch version: 0.4.1
GPU benchmark: 8 x 1080 Ti
CUDA version: 9.0

Experiment config:
batch size: 16
num workers: 16
input size: 480x480

When I use the Base_OC_Module on the ADE20K dataset, my experiment will stop at a certain iteration without other notion output. And the utilization rate of GPU will drop to 0. Did you have the similar experience?

@PkuRainBow
Copy link
Collaborator

Sorry, I think it can be a bug of inplace-abn. Please refer to mapillary/inplace_abn#58.

We use 4 x P100 for all the experiments.

@yu-changqian
Copy link
Author

Yes. When I use 4 x 1080Ti, all of it becomes normal. This should be a bug of inplace-abn. Thank you.

@KeyKy
Copy link

KeyKy commented Oct 9, 2018

I get the same problem and it only work for 4 gpus.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants