Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add lazy-resampling benchmark tutorial #1322

Merged
merged 14 commits into from
Apr 26, 2023
Merged

Conversation

KumoLiu
Copy link
Contributor

@KumoLiu KumoLiu commented Apr 20, 2023

part of Project-MONAI/MONAI#6156

Description

Show lazy-resampling benchmark results on 3d spleen segmentation. The results include the following two main parts.

  • Time spent on each transform and the total amount of data preparation in the lazy and non-lazy mode.
  • End-to-end time comparison

Checks

  • Avoid including large-size files in the PR.
  • Clean up long text outputs from code cells in the notebook.
  • For security purposes, please check the contents and remove any sensitive info such as user names and private key.
  • Ensure (1) hyperlinks and markdown anchors are working (2) use relative paths for tutorial repo files (3) put figure and graphs in the ./figure folder
  • Notebook runs automatically ./runner.sh -t <path to .ipynb file>

Signed-off-by: KumoLiu <yunl@nvidia.com>
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

pre-commit-ci bot and others added 11 commits April 20, 2023 07:00
Signed-off-by: KumoLiu <yunl@nvidia.com>
Signed-off-by: KumoLiu <yunl@nvidia.com>
Signed-off-by: KumoLiu <yunl@nvidia.com>
Signed-off-by: KumoLiu <yunl@nvidia.com>
Signed-off-by: KumoLiu <yunl@nvidia.com>
Signed-off-by: KumoLiu <yunl@nvidia.com>
Signed-off-by: KumoLiu <yunl@nvidia.com>
@KumoLiu KumoLiu marked this pull request as ready for review April 26, 2023 02:50
@KumoLiu
Copy link
Contributor Author

KumoLiu commented Apr 26, 2023

Hi @wyli, @Nic-Ma, since I was using latest MONAI version to do the benchmark, can I just use MONAI 1.2.0 in the tutorial or should I use 1.2.0rc5 and updated it once 1.2.0 release?

@KumoLiu KumoLiu requested a review from Nic-Ma April 26, 2023 02:52
@wyli
Copy link
Contributor

wyli commented Apr 26, 2023

Hi @wyli, @Nic-Ma, since I was using latest MONAI version to do the benchmark, can I just use MONAI 1.2.0 in the tutorial or should I use 1.2.0rc5 and updated it once 1.2.0 release?

The main goal is to make the numbers reported reproducible as much as possible, so I think we just report what has been used in the experiments. 1.2.0rc5. (and pytorch 1.14.0? I couldn't find it)

@KumoLiu
Copy link
Contributor Author

KumoLiu commented Apr 26, 2023

Hi @wyli, @Nic-Ma, since I was using latest MONAI version to do the benchmark, can I just use MONAI 1.2.0 in the tutorial or should I use 1.2.0rc5 and updated it once 1.2.0 release?

The main goal is to make the numbers reported reproducible as much as possible, so I think we just report what has been used in the experiments. 1.2.0rc5. (and pytorch 1.14.0? I couldn't find it)

Hi @wyli, but I'm not test on 1.2.0rc5 because we have some new changes such as Project-MONAI/MONAI#6412, and the pytorch version is "1.14.0a0+410ce96". Maybe I should change both version to the stable one and retrain it?

@wyli
Copy link
Contributor

wyli commented Apr 26, 2023

Hi @wyli, @Nic-Ma, since I was using latest MONAI version to do the benchmark, can I just use MONAI 1.2.0 in the tutorial or should I use 1.2.0rc5 and updated it once 1.2.0 release?

The main goal is to make the numbers reported reproducible as much as possible, so I think we just report what has been used in the experiments. 1.2.0rc5. (and pytorch 1.14.0? I couldn't find it)

Hi @wyli, but I'm not test on 1.2.0rc5 because we have some new changes such as Project-MONAI/MONAI#6412, and the pytorch version is "1.14.0a0+410ce96". Maybe I should change both version to the stable one and retrain it?

sure, you can also just report the monai version from python -c "import monai; monai.config.print_config()" I see in the notebook there is 1.2.0rc4+9.ge18097d0 and 1.13.0a0+d0d6b1f these are good enough for reproducing the results.

Signed-off-by: KumoLiu <yunl@nvidia.com>
@wyli wyli merged commit 97d0af4 into Project-MONAI:main Apr 26, 2023
@KumoLiu KumoLiu deleted the lazy-benchmark branch April 26, 2023 09:50
boneseva pushed a commit to boneseva/MONAI-tutorials that referenced this pull request Apr 21, 2024
part of Project-MONAI/MONAI#6156

### Description
Show lazy-resampling benchmark results on 3d spleen segmentation. The
results include the following two main parts.
- Time spent on each transform and the total amount of data preparation
in the lazy and non-lazy mode.
- End-to-end time comparison

### Checks
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Avoid including large-size files in the PR.
- [ ] Clean up long text outputs from code cells in the notebook.
- [ ] For security purposes, please check the contents and remove any
sensitive info such as user names and private key.
- [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use
relative paths for tutorial repo files (3) put figure and graphs in the
`./figure` folder
- [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>`

---------

Signed-off-by: KumoLiu <yunl@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants