Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bundle script - pprint doesn't take care of long list correctly #5762

Closed
SachidanandAlle opened this issue Dec 16, 2022 · 0 comments · Fixed by #5969
Closed

bundle script - pprint doesn't take care of long list correctly #5762

SachidanandAlle opened this issue Dec 16, 2022 · 0 comments · Fixed by #5969
Assignees
@wyli
Labels
enhancement New feature or request

Comments

@SachidanandAlle
Copy link
Contributor

Describe the bug
All the key/values are getting print here. But when the value is a huge list, the print/log is too big
https://github.com/Project-MONAI/MONAI/blob/dev/monai/bundle/scripts.py#L84

Looks like the pprint.pformat doesn't handle large lists correct.

To Reproduce
Steps to reproduce the behavior:

  1. You can use any monai bundle and pass the list of image/value pairs to the train/validation dataset
  2. Run single gpu training

Expected behavior
Smaller (first few elements and ...) log for the list; And if possible len of the list. Same should be in case of dict.

Screenshots
Currently it prints something huge.. like

2022-12-16 00:30:00,174 - INFO - > validate#dataset#data: [{'image': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_039_image.npy',
  'label_inst': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_039_inst_map.npy',
  'label_type': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_039_type_map.npy'},
 {'image': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_040_image.npy',
  'label_inst': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_040_inst_map.npy',
  'label_type': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_040_type_map.npy'},
 {'image': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_041_image.npy',
  'label_inst': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_041_inst_map.npy',
  'label_type': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_041_type_map.npy'},
 {'image': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_042_image.npy',
  'label_inst': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_042_inst_map.npy',
  'label_type': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_042_type_map.npy'},
 {'image': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_043_image.npy',
  'label_inst': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_043_inst_map.npy',
  'label_type': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_043_type_map.npy'},
 {'image': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_044_image.npy',
  'label_inst': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_044_inst_map.npy',
  'label_type': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_044_type_map.npy'},
 {'image': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_045_image.npy',
  'label_inst': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_045_inst_map.npy',
  'label_type': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_045_type_map.npy'},
 {'image': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_046_image.npy',
  'label_inst': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_046_inst_map.npy',
  'label_type': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_046_type_map.npy'},
 {'image': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_047_image.npy',
  'label_inst': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_047_inst_map.npy',
  'label_type': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_047_type_map.npy'},
 {'image': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_048_image.npy',
  'label_inst': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_048_inst_map.npy',
  'label_type': '/localhome/sachi/Projects/MONAILabel/sample-apps/pathology/model/pathology_nuclei_segmentation_classification/cache/train_ds/nuclei_hovernet/test_10_0_0_1000_1000_0x0_048_type_map.npy'}]

Environment

Ensuring you use the relevant python executable, please paste the output of:

================================
Printing MONAI config...
================================
MONAI version: 1.1.0rc2
Numpy version: 1.23.4
Pytorch version: 1.13.0+cu116
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: d8652642d7ae849ccf7f53cbb483344682199043
MONAI __file__: /localhome/sachi/.local/lib/python3.10/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.10
Nibabel version: 4.0.2
scikit-image version: 0.19.3
Pillow version: 9.0.1
Tensorboard version: 2.10.1
gdown version: 4.5.3
TorchVision version: 0.14.0+cu116
tqdm version: 4.64.1
lmdb version: 1.3.0
psutil version: 5.9.3
pandas version: 1.5.1
einops version: 0.5.0
transformers version: 4.21.3
mlflow version: 2.0.1
pynrrd version: 0.4.3

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04 LTS
Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
Processor: x86_64
Machine: x86_64
Python version: 3.10.6
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 16
Num logical CPUs: 32
Num usable CPUs: 32
CPU usage (%): [5.4, 5.4, 5.1, 6.5, 7.1, 5.4, 6.4, 5.7, 5.1, 6.8, 5.4, 5.1, 6.1, 6.8, 5.4, 5.1, 6.7, 4.7, 6.1, 5.4, 5.1, 6.1, 5.4, 5.1, 5.1, 4.7, 5.4, 5.1, 5.7, 5.4, 4.7, 99.3]
CPU freq. (MHz): 1062
Load avg. in last 1, 5, 15 mins (%): [8.5, 3.7, 3.9]
Disk usage (%): 4.7
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 251.6
Available memory (GB): 236.0
Used memory (GB): 13.5

================================
Printing GPU config...
================================
Num GPUs: 2
Has CUDA: True
CUDA version: 11.6
cuDNN enabled: True
cuDNN version: 8302
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']
GPU 0 Name: NVIDIA A40
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 84
GPU 0 Total memory (GB): 44.4
GPU 0 CUDA capability (maj.min): 8.6
GPU 1 Name: NVIDIA A40
GPU 1 Is integrated: False
GPU 1 Is multi GPU board: False
GPU 1 Multi processor count: 84
GPU 1 Total memory (GB): 44.4
GPU 1 CUDA capability (maj.min): 8.6

Additional context
Add any other context about the problem here.
@wyli wyli added the enhancement New feature or request label Dec 16, 2022
@wyli wyli mentioned this issue Feb 9, 2023
Merged
7 tasks
@wyli wyli self-assigned this Feb 9, 2023
wyli added a commit that referenced this issue Feb 10, 2023
@wyli
5762 pprint head and tail bundle script (#5969)
5657b8f 
Signed-off-by: Wenqi Li <wenqil@nvidia.com>

Fixes #5762

### Description

limiting the number of printing lines

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [x] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [x] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: Wenqi Li <wenqil@nvidia.com>
wyli added a commit to wyli/MONAI that referenced this issue Feb 13, 2023
@wyli
5762 pprint head and tail bundle script (Project-MONAI#5969)
1130539 
Signed-off-by: Wenqi Li <wenqil@nvidia.com>

Fixes Project-MONAI#5762

### Description

limiting the number of printing lines

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [x] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [x] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wyli wyli

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

5762 pprint head and tail bundle script wyli/MONAI
2 participants
@wyli @SachidanandAlle