Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Insufficient GPU memory #10

Open
sunck1 opened this issue Jun 9, 2023 · 4 comments
Open

Insufficient GPU memory #10

sunck1 opened this issue Jun 9, 2023 · 4 comments

Comments

@sunck1
Copy link

sunck1 commented Jun 9, 2023

Dear authors, I encountered an insufficient GPU memory problem, when I trained the VQGAN model based on Kits19 dataset. I found you trained this medical diffusion model with a 24GB memory GPU. According to your default parameter setting, I failed to train the VQGAN model( n_codes = 16384) on a 50GB memory GPU (although the batch size = 1). It seems like this model can be adopted to process the whole CT scan without cropping. What should I do to cope with this problem, not affecting the performance of your model? I would appreciate if you can give me a hand!!

@WhenMelancholy
Copy link

I encountered the same problem as well. When training on the A800 with 80GB of VRAM, I also experience out-of-memory issues.

@WhenMelancholy
Copy link

Dear authors, I encountered an insufficient GPU memory problem, when I trained the VQGAN model based on Kits19 dataset. I found you trained this medical diffusion model with a 24GB memory GPU. According to your default parameter setting, I failed to train the VQGAN model( n_codes = 16384) on a 50GB memory GPU (although the batch size = 1). It seems like this model can be adopted to process the whole CT scan without cropping. What should I do to cope with this problem, not affecting the performance of your model? I would appreciate if you can give me a hand!!

I have fixed this issue, and the modifications I made are as follows:

  1. I downgraded the PyTorch version from 2.0 to 1.20 to match the version used by the author.
  2. I modified the "downsample" parameter to be consistent with the supplementary materials of the paper.

@TWWinde
Copy link

TWWinde commented May 19, 2024

Dear authors, I encountered an insufficient GPU memory problem, when I trained the VQGAN model based on Kits19 dataset. I found you trained this medical diffusion model with a 24GB memory GPU. According to your default parameter setting, I failed to train the VQGAN model( n_codes = 16384) on a 50GB memory GPU (although the batch size = 1). It seems like this model can be adopted to process the whole CT scan without cropping. What should I do to cope with this problem, not affecting the performance of your model? I would appreciate if you can give me a hand!!

I have fixed this issue, and the modifications I made are as follows:

  1. I downgraded the PyTorch version from 2.0 to 1.20 to match the version used by the author.
  2. I modified the "downsample" parameter to be consistent with the supplementary materials of the paper.

I also encounter this Problem, could you explain it more in detail. Thank you so much for your help!!

@script-Yang
Copy link

Dear authors, I encountered an insufficient GPU memory problem, when I trained the VQGAN model based on Kits19 dataset. I found you trained this medical diffusion model with a 24GB memory GPU. According to your default parameter setting, I failed to train the VQGAN model( n_codes = 16384) on a 50GB memory GPU (although the batch size = 1). It seems like this model can be adopted to process the whole CT scan without cropping. What should I do to cope with this problem, not affecting the performance of your model? I would appreciate if you can give me a hand!!

I have fixed this issue, and the modifications I made are as follows:

  1. I downgraded the PyTorch version from 2.0 to 1.20 to match the version used by the author.
  2. I modified the "downsample" parameter to be consistent with the supplementary materials of the paper.

I also encounter this Problem, could you explain it more in detail. Thank you so much for your help!!

I recently encountered this problem when reproducing the experiment on the Breast MRI dataset. The way I solved it was to replace the default parameter model.downsample=[2,2,2] in the command with model.downsample=[4,4,4]. This is also the recommended parameter mentioned by the author in the supplementary material (specifically for the Brats dataset).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants