Adding support for Multi-GPU data analyzer #6182 #6202

heyufan1995 · 2023-03-20T17:57:57Z

Fixes #6182 .
fixes #6114
Added multi-gpu support for data analyzer.
Tested on NGC using 4 16g V100 for total segmentator
Speed up:
4 GPU: data analysis 6.90 mins (9.2 mins including yaml export)
1 GPU: data analysis 22.98 mins (25.33 mins including yaml export)

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

for more information, see https://pre-commit.ci

myron · 2023-03-20T23:26:49Z

Hi @heyufan1995 it seems you did your changes on top of an old commit (not the latest), because you changes remove the most recent logic of saving 2 files. Please double check , thanks.

monai/apps/auto3dseg/data_analyzer.py

mingxin-zheng · 2023-03-21T03:07:19Z

The signature is missing so it's not passing the DCO. For reference: https://github.com/Project-MONAI/MONAI/blob/dev/CONTRIBUTING.md#signing-your-work

monai/apps/auto3dseg/data_analyzer.py

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

for more information, see https://pre-commit.ci

monai/apps/auto3dseg/data_analyzer.py

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

wyli · 2023-03-31T17:56:03Z

/black

wyli · 2023-03-31T18:03:17Z

/build

wyli · 2023-04-02T09:17:02Z

/black
there are some testing errors
https://github.com/Project-MONAI/MONAI/actions/runs/4577850785/jobs/8083767762?pr=6202
for cpuonly tests and command python -m tests.test_auto3dseg.

mingxin-zheng · 2023-04-03T13:45:41Z

I just checked the "cpuonly" tests is not using "cpu" as device after one PR that changes the default device. I have a PR to fix that.

monai/apps/auto3dseg/data_analyzer.py

mingxin-zheng · 2023-04-03T13:57:14Z

By checking the error, it seems the cpu test fails because label is unreferenced. So it is not related to PR #6278 (but it will be good to include that fix too so that we can expose the bug earlier.)

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

Signed-off-by: Wenqi Li <831580+wyli@users.noreply.github.com>

for more information, see https://pre-commit.ci

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

monai/apps/auto3dseg/data_analyzer.py

wyli · 2023-04-05T19:21:42Z

/black

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

wyli · 2023-04-05T20:23:05Z

/build

wyli

merging this if the existing tests are ok. #6307 requires multi-gpu test environment which is under development.

wyli · 2023-04-05T21:12:39Z

some issues with

torch==1.13.1 torchvision==0.14.1

[2023-04-05T21:08:47.173Z] ======================================================================
[2023-04-05T21:08:47.173Z] ERROR: test_get_history (tests.test_auto3dseg_hpo.TestHPO)
[2023-04-05T21:08:47.173Z] ----------------------------------------------------------------------
[2023-04-05T21:08:47.173Z] Traceback (most recent call last):
[2023-04-05T21:08:47.173Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_auto3dseg_hpo.py", line 133, in setUp
[2023-04-05T21:08:47.173Z]     bundle_generator.generate(work_dir, num_fold=1)
[2023-04-05T21:08:47.173Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/apps/auto3dseg/bundle_gen.py", line 537, in generate
[2023-04-05T21:08:47.173Z]     gen_algo.export_to_disk(output_folder, name, fold=f_id)
[2023-04-05T21:08:47.173Z]   File "/tmp/tmpconkvfbv/workdir/algorithm_templates/segresnet/scripts/algo.py", line 250, in export_to_disk
[2023-04-05T21:08:47.173Z]     super().export_to_disk(output_path=output_path, algo_name=algo_name, **kwargs)
[2023-04-05T21:08:47.173Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/apps/auto3dseg/bundle_gen.py", line 151, in export_to_disk
[2023-04-05T21:08:47.173Z]     self.fill_records = self.fill_template_config(self.data_stats_files, self.output_path, **kwargs)
[2023-04-05T21:08:47.173Z]   File "/tmp/tmpconkvfbv/workdir/algorithm_templates/segresnet/scripts/algo.py", line 127, in fill_template_config
[2023-04-05T21:08:47.173Z]     max_epochs = int(np.clip(np.ceil(80000.0 / n_cases), a_min=300, a_max=1250))
[2023-04-05T21:08:47.173Z] ZeroDivisionError: float division by zero
[2023-04-05T21:08:47.173Z] 
[2023-04-05T21:08:47.173Z] ======================================================================
[2023-04-05T21:08:47.173Z] ERROR: test_run_algo (tests.test_auto3dseg_hpo.TestHPO)
[2023-04-05T21:08:47.173Z] ----------------------------------------------------------------------
[2023-04-05T21:08:47.173Z] Traceback (most recent call last):
[2023-04-05T21:08:47.173Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_auto3dseg_hpo.py", line 133, in setUp
[2023-04-05T21:08:47.173Z]     bundle_generator.generate(work_dir, num_fold=1)
[2023-04-05T21:08:47.173Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/apps/auto3dseg/bundle_gen.py", line 537, in generate
[2023-04-05T21:08:47.173Z]     gen_algo.export_to_disk(output_folder, name, fold=f_id)
[2023-04-05T21:08:47.173Z]   File "/tmp/tmpconkvfbv/workdir/algorithm_templates/segresnet/scripts/algo.py", line 250, in export_to_disk
[2023-04-05T21:08:47.173Z]     super().export_to_disk(output_path=output_path, algo_name=algo_name, **kwargs)
[2023-04-05T21:08:47.173Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/apps/auto3dseg/bundle_gen.py", line 151, in export_to_disk
[2023-04-05T21:08:47.173Z]     self.fill_records = self.fill_template_config(self.data_stats_files, self.output_path, **kwargs)
[2023-04-05T21:08:47.173Z]   File "/tmp/tmpconkvfbv/workdir/algorithm_templates/segresnet/scripts/algo.py", line 127, in fill_template_config
[2023-04-05T21:08:47.173Z]     max_epochs = int(np.clip(np.ceil(80000.0 / n_cases), a_min=300, a_max=1250))
[2023-04-05T21:08:47.173Z] ZeroDivisionError: float division by zero
[2023-04-05T21:08:47.173Z] 
[2023-04-05T21:08:47.173Z] ======================================================================
[2023-04-05T21:08:47.173Z] ERROR: test_run_optuna (tests.test_auto3dseg_hpo.TestHPO)
[2023-04-05T21:08:47.173Z] ----------------------------------------------------------------------
[2023-04-05T21:08:47.173Z] Traceback (most recent call last):
[2023-04-05T21:08:47.173Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_auto3dseg_hpo.py", line 133, in setUp
[2023-04-05T21:08:47.173Z]     bundle_generator.generate(work_dir, num_fold=1)
[2023-04-05T21:08:47.173Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/apps/auto3dseg/bundle_gen.py", line 537, in generate
[2023-04-05T21:08:47.173Z]     gen_algo.export_to_disk(output_folder, name, fold=f_id)
[2023-04-05T21:08:47.173Z]   File "/tmp/tmpconkvfbv/workdir/algorithm_templates/segresnet/scripts/algo.py", line 250, in export_to_disk
[2023-04-05T21:08:47.173Z]     super().export_to_disk(output_path=output_path, algo_name=algo_name, **kwargs)
[2023-04-05T21:08:47.173Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/apps/auto3dseg/bundle_gen.py", line 151, in export_to_disk
[2023-04-05T21:08:47.173Z]     self.fill_records = self.fill_template_config(self.data_stats_files, self.output_path, **kwargs)
[2023-04-05T21:08:47.173Z]   File "/tmp/tmpconkvfbv/workdir/algorithm_templates/segresnet/scripts/algo.py", line 127, in fill_template_config
[2023-04-05T21:08:47.173Z]     max_epochs = int(np.clip(np.ceil(80000.0 / n_cases), a_min=300, a_max=1250))
[2023-04-05T21:08:47.173Z] ZeroDivisionError: float division by zero
[2023-04-05T21:08:47.173Z] 
[2023-04-05T21:08:47.173Z] ----------------------------------------------------------------------

monai/apps/auto3dseg/data_analyzer.py

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

wyli · 2023-04-11T20:17:11Z

/black

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

wyli · 2023-04-11T20:35:31Z

/integration-test
/build

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

wyli · 2023-04-11T21:59:45Z

/build

heyufan1995 and others added 2 commits March 17, 2023 11:26

Add multi-gpu data analyzer

46a7725

Merge branch 'Project-MONAI:dev' into dev

14f1d4e

heyufan1995 assigned wyli and dongyang0122 Mar 20, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

6f4a0cc

for more information, see https://pre-commit.ci

wyli requested review from mingxin-zheng, wyli and Nic-Ma March 20, 2023 19:23

heyufan1995 assigned mingxin-zheng and myron Mar 20, 2023

mingxin-zheng reviewed Mar 21, 2023

View reviewed changes

monai/apps/auto3dseg/data_analyzer.py Outdated Show resolved Hide resolved

mingxin-zheng reviewed Mar 21, 2023

View reviewed changes

monai/apps/auto3dseg/data_analyzer.py Outdated Show resolved Hide resolved

mingxin-zheng reviewed Mar 21, 2023

View reviewed changes

monai/apps/auto3dseg/data_analyzer.py Outdated Show resolved Hide resolved

mingxin-zheng reviewed Mar 21, 2023

View reviewed changes

monai/apps/auto3dseg/data_analyzer.py Show resolved Hide resolved

Update multi-gpu data analyzer

f748927

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

heyufan1995 force-pushed the dev branch from 9e8aa98 to f748927 Compare March 22, 2023 02:46

[pre-commit.ci] auto fixes from pre-commit.com hooks

0b71a76

for more information, see https://pre-commit.ci

wyli reviewed Mar 22, 2023

View reviewed changes

monai/apps/auto3dseg/data_analyzer.py Outdated Show resolved Hide resolved

heyufan1995 and others added 4 commits March 23, 2023 17:11

Merge branch 'Project-MONAI:dev' into dev

d76feff

Merge branch 'Project-MONAI:dev' into dev

0c28435

Ignore non-json/yaml configs for BundleAlg

e556ec2

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

Skip multiprocessing init with cpu/single gpu

033f14d

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

heyufan1995 force-pushed the dev branch from 2bd7848 to 033f14d Compare March 27, 2023 19:07

Remove warning filter

a9689fa

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

mingxin-zheng reviewed Apr 3, 2023

View reviewed changes

monai/apps/auto3dseg/data_analyzer.py Show resolved Hide resolved

Fix label reference error for data analyzer

e2ebaac

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

heyufan1995 force-pushed the dev branch from 498755b to e2ebaac Compare April 3, 2023 15:16

heyufan1995 and others added 5 commits April 3, 2023 11:20

delete a tmp file

c39e904

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

Merge branch 'dev' into dev

50d5633

Signed-off-by: Wenqi Li <831580+wyli@users.noreply.github.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

c536d44

for more information, see https://pre-commit.ci

[MONAI] code formatting

5e58527

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

Update data_analyzer.py

7a0b73b

wyli reviewed Apr 5, 2023

View reviewed changes

monai/apps/auto3dseg/data_analyzer.py Outdated Show resolved Hide resolved

wyli added 2 commits April 5, 2023 19:45

Update data_analyzer.py

8b6c6be

Merge branch 'dev' into dev

6a4ea86

monai-bot and others added 2 commits April 5, 2023 19:27

[MONAI] code formatting

64148e9

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

Update data_analyzer.py

41645e1

wyli mentioned this pull request Apr 5, 2023

unit/integration tests for multi-gpu data analyzer #6307

Closed

Merge branch 'dev' into dev

79b3c6a

wyli approved these changes Apr 5, 2023

View reviewed changes

myron reviewed Apr 6, 2023

View reviewed changes

monai/apps/auto3dseg/data_analyzer.py Show resolved Hide resolved

monai/apps/auto3dseg/data_analyzer.py Outdated Show resolved Hide resolved

monai/apps/auto3dseg/data_analyzer.py Outdated Show resolved Hide resolved

heyufan1995 and others added 2 commits April 11, 2023 16:09

change image_only to True for dataanalyzer

d52e50f

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

Merge branch 'dev' into dev

1918f15

[MONAI] code formatting

35dc2bd

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

Add stats by case back to datastats.yaml

52d0d40

Signed-off-by: heyufan1995 <heyufan1995@gmail.com>

wyli enabled auto-merge (squash) April 11, 2023 22:36

wyli merged commit 6a7f35b into Project-MONAI:dev Apr 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding support for Multi-GPU data analyzer #6182 #6202

Adding support for Multi-GPU data analyzer #6182 #6202

heyufan1995 commented Mar 20, 2023 •

edited by wyli

Loading

myron commented Mar 20, 2023

mingxin-zheng commented Mar 21, 2023

wyli commented Mar 31, 2023

wyli commented Mar 31, 2023

wyli commented Apr 2, 2023 •

edited

Loading

mingxin-zheng commented Apr 3, 2023

mingxin-zheng commented Apr 3, 2023

wyli commented Apr 5, 2023

wyli commented Apr 5, 2023

wyli left a comment

wyli commented Apr 5, 2023 •

edited

Loading

wyli commented Apr 11, 2023

wyli commented Apr 11, 2023 •

edited

Loading

wyli commented Apr 11, 2023

Adding support for Multi-GPU data analyzer #6182 #6202

Adding support for Multi-GPU data analyzer #6182 #6202

Conversation

heyufan1995 commented Mar 20, 2023 • edited by wyli Loading

Types of changes

myron commented Mar 20, 2023

mingxin-zheng commented Mar 21, 2023

wyli commented Mar 31, 2023

wyli commented Mar 31, 2023

wyli commented Apr 2, 2023 • edited Loading

mingxin-zheng commented Apr 3, 2023

mingxin-zheng commented Apr 3, 2023

wyli commented Apr 5, 2023

wyli commented Apr 5, 2023

wyli left a comment

Choose a reason for hiding this comment

wyli commented Apr 5, 2023 • edited Loading

wyli commented Apr 11, 2023

wyli commented Apr 11, 2023 • edited Loading

wyli commented Apr 11, 2023

heyufan1995 commented Mar 20, 2023 •

edited by wyli

Loading

wyli commented Apr 2, 2023 •

edited

Loading

wyli commented Apr 5, 2023 •

edited

Loading

wyli commented Apr 11, 2023 •

edited

Loading