Adding Tailored ControlNet Implementations into Generative Model Application #7875

guopengf · 2024-06-24T21:36:03Z

Description

Integrating a tailored ControlNet model into the generative model application to enable the training using high-dimensional 3D images (up to 512 x 512 x 768).

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

monai/apps/generation/maisi/networks/controlnet_maisi.py

tests/test_controlnet_maisi.py

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

for more information, see https://pre-commit.ci Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

…ct-MONAI#7847) Fixes Project-MONAI#7677 . ### Description A few sentences describing the changes proposed in this pull request. ### Types of changes  - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. --------- Signed-off-by: alkamid <adam.klimont@cydar.co.uk> Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Pengfei Guo <32000655+guopengf@users.noreply.github.com> Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

KumoLiu · 2024-06-27T13:47:25Z

Hi @ericspod @Nic-Ma @mingxin-zheng, if you don't have more comments on this one, I will try to trigger more tests and merge this one. Thanks!

KumoLiu · 2024-06-28T06:27:21Z

Hi @guopengf, could you please help solve the conflict then I will try to merge this PR, thanks!

Signed-off-by: Pengfei Guo <32000655+guopengf@users.noreply.github.com>

guopengf · 2024-06-28T14:50:51Z

Hi @guopengf, could you please help solve the conflict then I will try to merge this PR, thanks!

Done, thanks!

KumoLiu · 2024-06-28T15:00:52Z

/build

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

ericspod

I'm good here!

KumoLiu · 2024-06-29T12:50:21Z

/build

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>

KumoLiu · 2024-07-01T05:39:35Z

/build

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>

KumoLiu · 2024-07-01T05:53:24Z

/build

KumoLiu · 2024-07-01T06:59:06Z

Add monai-generative, will cause this error:


[2024-06-29T13:15:38.518Z] ======================================================================

[2024-06-29T13:15:38.518Z] FAIL: test_visible_devices (tests.test_set_visible_devices.TestVisibleDevices)

[2024-06-29T13:15:38.518Z] ----------------------------------------------------------------------

[2024-06-29T13:15:38.518Z] Traceback (most recent call last):

[2024-06-29T13:15:38.518Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_set_visible_devices.py", line 38, in test_visible_devices

[2024-06-29T13:15:38.518Z]     self.assertEqual(num_gpus_before, num_gpus_after)

[2024-06-29T13:15:38.518Z] AssertionError: 0 != 1

[2024-06-29T13:15:38.518Z] 

[2024-06-29T13:15:38.518Z] ----------------------------------------------------------------------

Do you have any insights on this one? @ericspod @Nic-Ma @mingxin-zheng

monai/apps/generation/maisi/networks/controlnet_maisi.py

mingxin-zheng · 2024-07-01T08:38:43Z

Do you have any insights on this one? @ericspod @Nic-Ma @mingxin-zheng

It seems the error is caused by this line:

ControlNet, has_controlnet = optional_import("generative.networks.nets.controlnet", name="ControlNet")

mingxin-zheng · 2024-07-01T09:11:14Z

Update: pinpointed that removing

@torch.cuda.amp.autocast

will solve the issue @KumoLiu

https://github.com/Project-MONAI/GenerativeModels/blob/a9b17d461d42b7b3487d0e29be265d9373b11d67/generative/networks/layers/vector_quantizer.py#L20

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>

KumoLiu · 2024-07-01T09:34:53Z

/build

for more information, see https://pre-commit.ci

mingxin-zheng

Looks good to me.

mingxin-zheng · 2024-07-01T13:39:26Z

I am merging this to unblock #7861.

guopengf self-assigned this Jun 24, 2024

KumoLiu reviewed Jun 25, 2024

View reviewed changes

monai/apps/generation/maisi/networks/controlnet_maisi.py Outdated Show resolved Hide resolved

KumoLiu reviewed Jun 25, 2024

View reviewed changes

monai/apps/generation/maisi/networks/controlnet_maisi.py Show resolved Hide resolved

guopengf force-pushed the fix-issue-7874-add-controlnet branch from 7e2a8b0 to 52428b0 Compare June 25, 2024 22:38

guopengf marked this pull request as ready for review June 26, 2024 02:34

KumoLiu requested review from ericspod and Nic-Ma June 26, 2024 03:15

KumoLiu reviewed Jun 26, 2024

View reviewed changes

monai/apps/generation/maisi/networks/controlnet_maisi.py Outdated Show resolved Hide resolved

tests/test_controlnet_maisi.py Show resolved Hide resolved

guopengf and others added 15 commits June 26, 2024 15:14

inital commit

9f27c15

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

63290b5

for more information, see https://pre-commit.ci Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

add unit test and refactor forward

15fb9f6

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

update

2ea4718

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

update

b191e7f

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

update

24c6fb2

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

update

63ca241

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

update

bca7aa2

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

update

6413581

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

update

8bdbc79

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

update

1daeddf

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

update

f992471

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

Update monai/apps/generation/maisi/networks/controlnet_maisi.py

9ae94fe

Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Pengfei Guo <32000655+guopengf@users.noreply.github.com> Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

add more test cases

c342b23

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

guopengf force-pushed the fix-issue-7874-add-controlnet branch from 3afc58d to c342b23 Compare June 26, 2024 19:15

Merge branch 'dev' into fix-issue-7874-add-controlnet

c671149

Merge branch 'dev' into fix-issue-7874-add-controlnet

3038865

Signed-off-by: Pengfei Guo <32000655+guopengf@users.noreply.github.com>

KumoLiu approved these changes Jun 28, 2024

View reviewed changes

guopengf added 3 commits June 28, 2024 12:51

update torch version req

6d24a51

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

update torch version req

5f85e3b

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

update pre-commit-config

403bf9b

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>

ericspod approved these changes Jun 28, 2024

View reviewed changes

Merge branch 'dev' into fix-issue-7874-add-controlnet

22e52b4

temp test

fcc95cc

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>

fix flake8

8eedb25

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>

mingxin-zheng reviewed Jul 1, 2024

View reviewed changes

monai/apps/generation/maisi/networks/controlnet_maisi.py Show resolved Hide resolved

temp-fix

984c5da

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

ebcf787

for more information, see https://pre-commit.ci

mingxin-zheng approved these changes Jul 1, 2024

View reviewed changes

mingxin-zheng mentioned this pull request Jul 1, 2024

Fix cuda initialize issue Project-MONAI/GenerativeModels#497

Merged

mingxin-zheng merged commit 55386e0 into Project-MONAI:dev Jul 1, 2024
28 checks passed

KumoLiu added this to the Refactor MAISI [P0 v1.4] milestone Aug 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Tailored ControlNet Implementations into Generative Model Application #7875

Adding Tailored ControlNet Implementations into Generative Model Application #7875

guopengf commented Jun 24, 2024

KumoLiu commented Jun 27, 2024

KumoLiu commented Jun 28, 2024

guopengf commented Jun 28, 2024

KumoLiu commented Jun 28, 2024

ericspod left a comment

KumoLiu commented Jun 29, 2024

KumoLiu commented Jul 1, 2024

KumoLiu commented Jul 1, 2024

KumoLiu commented Jul 1, 2024

mingxin-zheng commented Jul 1, 2024

mingxin-zheng commented Jul 1, 2024 •

edited

Loading

KumoLiu commented Jul 1, 2024

mingxin-zheng left a comment

mingxin-zheng commented Jul 1, 2024

Adding Tailored ControlNet Implementations into Generative Model Application #7875

Adding Tailored ControlNet Implementations into Generative Model Application #7875

Conversation

guopengf commented Jun 24, 2024

Description

Types of changes

KumoLiu commented Jun 27, 2024

KumoLiu commented Jun 28, 2024

guopengf commented Jun 28, 2024

KumoLiu commented Jun 28, 2024

ericspod left a comment

Choose a reason for hiding this comment

KumoLiu commented Jun 29, 2024

KumoLiu commented Jul 1, 2024

KumoLiu commented Jul 1, 2024

KumoLiu commented Jul 1, 2024

mingxin-zheng commented Jul 1, 2024

mingxin-zheng commented Jul 1, 2024 • edited Loading

KumoLiu commented Jul 1, 2024

mingxin-zheng left a comment

Choose a reason for hiding this comment

mingxin-zheng commented Jul 1, 2024

mingxin-zheng commented Jul 1, 2024 •

edited

Loading