auto3dseg analyzer incorrectly computes size of images, rounds to integers

[jpsacha]

Describe the bug
The auto3dseg analyzer when computing sizes of images is converting the size in mm to integers. For small images, like OCT, that can results in image size of [0, 0, 0] and issues in code that is using the results produced by the analyzer.
The incorrect code is here:
https://github.com/Project-MONAI/MONAI/blame/782c1e616312612c9f25bf4a33986ace67f7851e/monai/auto3dseg/analyzer.py#L259

To Reproduce
Consider an OCT image with spacing [0.001, 0.001, 0.001] and shape [308, 250, 256]. The correct image size in mm should be [0.308, 0.250, 0.256]. This can be reproduced with synthetic data too.

Steps to reproduce the behavior:

1. clone https://github.com/Project-MONAI/tutorials
2. Start jupyter
3. Go to notebooks/auto3dseg/notebooks/auto3dseg_hello_world.ipynb
4. Introduce spacing less than 1mm in the synthetic data. Go to section "Generate images and labels" and change spacing from 1mm to a fraction of 1mm. Change:

       nib.save(nib.Nifti1Image(im, affine=np.eye(4)), image_fpath)
       nib.save(nib.Nifti1Image(seg, affine=np.eye(4)), label_fpath)

   to

       nib.save(nib.Nifti1Image(im, affine=np.eye(4)*0.001), image_fpath)
       nib.save(nib.Nifti1Image(seg, affine=np.eye(4)*0.001), label_fpath)

5. Run the notebook. In section "Start the data analysis, algorithm generation, training, and model ensemble" you will get a misleading error:

2023-11-10 21:56:14,716 - INFO - Running data analysis...
2023-11-10 21:56:14,772 - INFO - Found 1 GPUs for data analyzing!

100%|██████████| 12/12 [00:01<00:00, 11.15it/s]

2023-11-10 21:56:16,107 - INFO - BundleGen from https://github.com/Project-MONAI/research-contributions/releases/download/algo_templates/9db946d.tar.gz

algo_templates.tar.gz: 96.0kB [00:00, 380kB/s]                             

2023-11-10 21:56:16,370 - INFO - Downloaded: /tmp/tmplkoq_ovq/algo_templates.tar.gz
2023-11-10 21:56:16,371 - INFO - Expected md5 is None, skip md5 check for file /tmp/tmplkoq_ovq/algo_templates.tar.gz.
2023-11-10 21:56:16,371 - INFO - Writing into directory: /x/MONAI_Tutorials/auto3dseg/notebooks/helloworld_work_dir.


2023-11-10 21:56:16,571 - INFO - Generated:/x/MONAI_Tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0

divide by zero encountered in log2
invalid value encountered in divide

---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
Cell In[8], line 1
----> 1 runner.run()

File ~/miniconda3/envs/monailabel/lib/python3.10/site-packages/monai/apps/auto3dseg/auto_runner.py:775, in AutoRunner.run(self)
    767     bundle_generator.generate(
    768         self.work_dir,
    769         num_fold=self.num_fold,
   (...)
    772         allow_skip=self.allow_skip,
    773     )
    774 else:
--> 775     bundle_generator.generate(self.work_dir, num_fold=self.num_fold, allow_skip=self.allow_skip)
    776 history = bundle_generator.get_history()
    777 export_bundle_algo_history(history)

File ~/miniconda3/envs/monailabel/lib/python3.10/site-packages/monai/apps/auto3dseg/bundle_gen.py:589, in BundleGen.generate(self, output_folder, num_fold, gpu_customization, gpu_customization_specs, allow_skip)
    581     gen_algo.export_to_disk(
    582         output_folder,
    583         name,
   (...)
    586         gpu_customization_specs=gpu_customization_specs,
    587     )
    588 else:
--> 589     gen_algo.export_to_disk(output_folder, name, fold=f_id)
    591 algo_to_pickle(gen_algo, template_path=algo.template_path)
    592 self.history.append(
    593     {AlgoKeys.ID: name, AlgoKeys.ALGO: gen_algo}
    594 )

File /x/MONAI_Tutorials/auto3dseg/notebooks/helloworld_work_dir/algorithm_templates/segresnet/scripts/algo.py:294, in SegresnetAlgo.export_to_disk(self, output_path, algo_name, **kwargs)
    293 def export_to_disk(self, output_path: str, algo_name: str, **kwargs):
--> 294     super().export_to_disk(output_path=output_path, algo_name=algo_name, **kwargs)
    296     output_path = os.path.join(output_path, algo_name)
    297     config = ConfigParser.load_config_file(os.path.join(output_path, "configs/hyper_parameters.yaml"))

File ~/miniconda3/envs/monailabel/lib/python3.10/site-packages/monai/apps/auto3dseg/bundle_gen.py:163, in BundleAlgo.export_to_disk(self, output_path, algo_name, **kwargs)
    161     self.output_path = str(self.template_path)
    162 if kwargs.pop("fill_template", True):
--> 163     self.fill_records = self.fill_template_config(self.data_stats_files, self.output_path, **kwargs)
    164 logger.info(f"Generated:{self.output_path}")

File /x/MONAI_Tutorials/auto3dseg/notebooks/helloworld_work_dir/algorithm_templates/segresnet/scripts/algo.py:221, in SegresnetAlgo.fill_template_config(self, data_stats_file, output_path, **kwargs)
    217 config["num_epochs"] = max_epochs
    219 ###########################################
--> 221 roi_size, levels, init_filters, batch_size = auto_adjust_network_settings(
    222     auto_scale_batch=input_config.get("auto_scale_batch", False),
    223     auto_scale_roi=input_config.get("auto_scale_roi", False),
    224     auto_scale_filters=input_config.get("auto_scale_filters", False),
    225     image_size_mm=config["image_size_mm_median"],
    226     spacing=config["resample_resolution"],
    227     anisotropic_scales=config["anisotropic_scales"],
    228     output_classes=config["output_classes"],
    229 )
    231 if input_config.get("roi_size", None):
    232     roi_size = input_config.get("roi_size", None)

File /x/MONAI_Tutorials/auto3dseg/notebooks/helloworld_work_dir/algorithm_templates/segresnet/scripts/utils.py:213, in auto_adjust_network_settings(auto_scale_roi, auto_scale_batch, auto_scale_filters, image_size_mm, spacing, output_classes, levels, anisotropic_scales, levels_limit, gpu_mem)
    208     if global_rank == 0:
    209         print(
    210             f"kept batch the same base_numel {base_numel},  gpu_factor {gpu_factor},  gpu_factor_init {gpu_factor_init}"
    211         )
--> 213 levels = int(levels)
    214 roi_size = roi_size.astype(int).tolist()
    216 if global_rank == 0:

OverflowError: cannot convert float infinity to integer

The error is a consequence of image_size_mm being incorrectly calculated as [0,0,0]

Expected behavior
image_size_mm should be computed correctly for pixel sizes that are a fraction of a mm (like 0.001). Notebook should execute without errors

Environment

Ensuring you use the relevant python executable, please paste the output of:

python -c 'import monai; monai.config.print_debug_info()'

================================
Printing MONAI config...
================================
MONAI version: 1.3.0
Numpy version: 1.26.0
Pytorch version: 2.1.0+cu121
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 865972f7a791bf7b42efbcd87c8402bd865b329e
MONAI __file__: /home/<username>/miniconda3/envs/monailabel/lib/python3.10/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.3.0
Nibabel version: 5.1.0
scikit-image version: 0.22.0
scipy version: 1.11.3
Pillow version: 10.1.0
Tensorboard version: 2.14.1
gdown version: 4.7.1
TorchVision version: 0.16.0+cu121
tqdm version: 4.66.1
lmdb version: 1.4.1
psutil version: 5.9.5
pandas version: 2.1.1
einops version: 0.7.0
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: 2.7.1
pynrrd version: 1.0.0
clearml version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04.3 LTS
Platform: Linux-6.2.0-36-generic-x86_64-with-glibc2.35
Processor: x86_64
Machine: x86_64
Python version: 3.10.13
...

Additional context
The reported problem is happening when using a real world 3D data from an OCT instrument, but can be easily reproduced using synthetic data as illustrated above.

[KumoLiu]

Hi @jpsacha, thanks for the reporting.

Is there a reason why we need to force the conversion to int when calculating sizemm?
Could you please help confirming it? cc @myron @mingxin-zheng
Thanks!

[vikashg]

closing as inactive

[jpsacha]

Was this issue fixed? The problem is still there.

[mingxin-zheng]

Hi @jpsacha , thanks for reopening this thread and the follow up.

It seems this line is causing the problem:

MONAI/monai/auto3dseg/analyzer.py

Line 259 in 8fa6931

int(a * b) for a, b in zip(report[ImageStatsKeys.SHAPE][0], report[ImageStatsKeys.SPACING])

Hi @myron , is there any concern to using floating point number here?

[myron]

I see, let's change it to be in float.