4 gpus works fine while 8 gpus causes memory leakage and runs slow?

[hszhao]

I tested sync inplace abn on 8 x GTX TITAN and 8 x TITAN Xp machine. In both case, using 4 gpus works well while memory leakage appears when 8 gpus are adopted.
gpu: GeForce GTX TITAN and TITAN Xp
cuda: 8.0
pytorch: 0.4.1
gcc: 5.2.0

[Spandan-Madan]

Did you mean your GCC is 5.2.0? Also, did you compile pytorch from source or used conda?

I have the same configs but get errors when trying to run.

[hszhao]

@Spandan-Madan , yes, gcc is 5.2.0. I use conda to install pytorch and test my own code with sync inplace abn incorporated.

[KeyKy]

I get the same problem.

[ackness]

same problem.
8 gpus died, 4 gpus works well, 6 gpus very slow :(

[wangjingbo1219]

same problem

[rotabulo]

@hszhao @Spandan-Madan @ackness @wangjingbo1219 @KeyKy we have released today a new version of our code that requires PyTorch v1.0 and supports multi-process synchronization. This solves problems with training on more than 4 Gpus.

[rotabulo]

Closing because the new version should have solved the issue.

[charlesCXK]

Thank you, the newest version of Inplace ABN solves the problem.