Closed
Description
Describe the bug
The auto3dseg analyzer when computing sizes of images is converting the size in mm to integers. For small images, like OCT, that can results in image size of [0, 0, 0] and issues in code that is using the results produced by the analyzer.
The incorrect code is here:
https://github.com/Project-MONAI/MONAI/blame/782c1e616312612c9f25bf4a33986ace67f7851e/monai/auto3dseg/analyzer.py#L259
To Reproduce
Consider an OCT image with spacing [0.001, 0.001, 0.001] and shape [308, 250, 256]. The correct image size in mm should be [0.308, 0.250, 0.256]. This can be reproduced with synthetic data too.
Steps to reproduce the behavior:
- clone https://github.com/Project-MONAI/tutorials
- Start
jupyter - Go to
notebooks/auto3dseg/notebooks/auto3dseg_hello_world.ipynb - Introduce spacing less than 1mm in the synthetic data. Go to section "Generate images and labels" and change spacing from 1mm to a fraction of 1mm. Change:
to
nib.save(nib.Nifti1Image(im, affine=np.eye(4)), image_fpath) nib.save(nib.Nifti1Image(seg, affine=np.eye(4)), label_fpath)
nib.save(nib.Nifti1Image(im, affine=np.eye(4)*0.001), image_fpath) nib.save(nib.Nifti1Image(seg, affine=np.eye(4)*0.001), label_fpath)
- Run the notebook. In section "Start the data analysis, algorithm generation, training, and model ensemble" you will get a misleading error:
2023-11-10 21:56:14,716 - INFO - Running data analysis...
2023-11-10 21:56:14,772 - INFO - Found 1 GPUs for data analyzing!
100%|██████████| 12/12 [00:01<00:00, 11.15it/s]
2023-11-10 21:56:16,107 - INFO - BundleGen from https://github.com/Project-MONAI/research-contributions/releases/download/algo_templates/9db946d.tar.gz
algo_templates.tar.gz: 96.0kB [00:00, 380kB/s]
2023-11-10 21:56:16,370 - INFO - Downloaded: /tmp/tmplkoq_ovq/algo_templates.tar.gz
2023-11-10 21:56:16,371 - INFO - Expected md5 is None, skip md5 check for file /tmp/tmplkoq_ovq/algo_templates.tar.gz.
2023-11-10 21:56:16,371 - INFO - Writing into directory: /x/MONAI_Tutorials/auto3dseg/notebooks/helloworld_work_dir.
2023-11-10 21:56:16,571 - INFO - Generated:/x/MONAI_Tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0
divide by zero encountered in log2
invalid value encountered in divide
---------------------------------------------------------------------------
OverflowError Traceback (most recent call last)
Cell In[8], line 1
----> 1 runner.run()
File ~/miniconda3/envs/monailabel/lib/python3.10/site-packages/monai/apps/auto3dseg/auto_runner.py:775, in AutoRunner.run(self)
767 bundle_generator.generate(
768 self.work_dir,
769 num_fold=self.num_fold,
(...)
772 allow_skip=self.allow_skip,
773 )
774 else:
--> 775 bundle_generator.generate(self.work_dir, num_fold=self.num_fold, allow_skip=self.allow_skip)
776 history = bundle_generator.get_history()
777 export_bundle_algo_history(history)
File ~/miniconda3/envs/monailabel/lib/python3.10/site-packages/monai/apps/auto3dseg/bundle_gen.py:589, in BundleGen.generate(self, output_folder, num_fold, gpu_customization, gpu_customization_specs, allow_skip)
581 gen_algo.export_to_disk(
582 output_folder,
583 name,
(...)
586 gpu_customization_specs=gpu_customization_specs,
587 )
588 else:
--> 589 gen_algo.export_to_disk(output_folder, name, fold=f_id)
591 algo_to_pickle(gen_algo, template_path=algo.template_path)
592 self.history.append(
593 {AlgoKeys.ID: name, AlgoKeys.ALGO: gen_algo}
594 )
File /x/MONAI_Tutorials/auto3dseg/notebooks/helloworld_work_dir/algorithm_templates/segresnet/scripts/algo.py:294, in SegresnetAlgo.export_to_disk(self, output_path, algo_name, **kwargs)
293 def export_to_disk(self, output_path: str, algo_name: str, **kwargs):
--> 294 super().export_to_disk(output_path=output_path, algo_name=algo_name, **kwargs)
296 output_path = os.path.join(output_path, algo_name)
297 config = ConfigParser.load_config_file(os.path.join(output_path, "configs/hyper_parameters.yaml"))
File ~/miniconda3/envs/monailabel/lib/python3.10/site-packages/monai/apps/auto3dseg/bundle_gen.py:163, in BundleAlgo.export_to_disk(self, output_path, algo_name, **kwargs)
161 self.output_path = str(self.template_path)
162 if kwargs.pop("fill_template", True):
--> 163 self.fill_records = self.fill_template_config(self.data_stats_files, self.output_path, **kwargs)
164 logger.info(f"Generated:{self.output_path}")
File /x/MONAI_Tutorials/auto3dseg/notebooks/helloworld_work_dir/algorithm_templates/segresnet/scripts/algo.py:221, in SegresnetAlgo.fill_template_config(self, data_stats_file, output_path, **kwargs)
217 config["num_epochs"] = max_epochs
219 ###########################################
--> 221 roi_size, levels, init_filters, batch_size = auto_adjust_network_settings(
222 auto_scale_batch=input_config.get("auto_scale_batch", False),
223 auto_scale_roi=input_config.get("auto_scale_roi", False),
224 auto_scale_filters=input_config.get("auto_scale_filters", False),
225 image_size_mm=config["image_size_mm_median"],
226 spacing=config["resample_resolution"],
227 anisotropic_scales=config["anisotropic_scales"],
228 output_classes=config["output_classes"],
229 )
231 if input_config.get("roi_size", None):
232 roi_size = input_config.get("roi_size", None)
File /x/MONAI_Tutorials/auto3dseg/notebooks/helloworld_work_dir/algorithm_templates/segresnet/scripts/utils.py:213, in auto_adjust_network_settings(auto_scale_roi, auto_scale_batch, auto_scale_filters, image_size_mm, spacing, output_classes, levels, anisotropic_scales, levels_limit, gpu_mem)
208 if global_rank == 0:
209 print(
210 f"kept batch the same base_numel {base_numel}, gpu_factor {gpu_factor}, gpu_factor_init {gpu_factor_init}"
211 )
--> 213 levels = int(levels)
214 roi_size = roi_size.astype(int).tolist()
216 if global_rank == 0:
OverflowError: cannot convert float infinity to integer
The error is a consequence of image_size_mm being incorrectly calculated as [0,0,0]
Expected behavior
image_size_mm should be computed correctly for pixel sizes that are a fraction of a mm (like 0.001). Notebook should execute without errors
Environment
Ensuring you use the relevant python executable, please paste the output of:
python -c 'import monai; monai.config.print_debug_info()'
================================
Printing MONAI config...
================================
MONAI version: 1.3.0
Numpy version: 1.26.0
Pytorch version: 2.1.0+cu121
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 865972f7a791bf7b42efbcd87c8402bd865b329e
MONAI __file__: /home/<username>/miniconda3/envs/monailabel/lib/python3.10/site-packages/monai/__init__.py
Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.3.0
Nibabel version: 5.1.0
scikit-image version: 0.22.0
scipy version: 1.11.3
Pillow version: 10.1.0
Tensorboard version: 2.14.1
gdown version: 4.7.1
TorchVision version: 0.16.0+cu121
tqdm version: 4.66.1
lmdb version: 1.4.1
psutil version: 5.9.5
pandas version: 2.1.1
einops version: 0.7.0
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: 2.7.1
pynrrd version: 1.0.0
clearml version: NOT INSTALLED or UNKNOWN VERSION.
For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04.3 LTS
Platform: Linux-6.2.0-36-generic-x86_64-with-glibc2.35
Processor: x86_64
Machine: x86_64
Python version: 3.10.13
...
Additional context
The reported problem is happening when using a real world 3D data from an OCT instrument, but can be easily reproduced using synthetic data as illustrated above.