Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add NVTX handlers #2765

Merged
merged 24 commits into from
Aug 17, 2021
Merged

Add NVTX handlers #2765

merged 24 commits into from
Aug 17, 2021

Conversation

drbeh
Copy link
Member

@drbeh drbeh commented Aug 13, 2021 •
edited
Loading

Fixes #2764

Description

This PR adds NVTX handlers to enable probing and profiling MONAI workflow through Ignite events. It attached nvtx marks and ranges to Ignite events.

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@drbeh
Add NVTX handlers
ae5e3de 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Merge branch 'dev' of github.com:Project-MONAI/MONAI into nvtx-handlers
e1e616c
@drbeh
Update imports
0e24c7c 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Update docs
4c87466 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Add events to rangepush and mark handlers
79ef17e 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Add unittest for nvtx handlers
db45c65 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh drbeh marked this pull request as ready for review August 13, 2021 18:06
@drbeh drbeh mentioned this pull request Aug 13, 2021
Merged
@drbeh
Exclude handler tests from min test
a2183eb 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh drbeh requested review from Nic-Ma and wyli August 13, 2021 18:18
@drbeh
Copy link
Member Author

drbeh commented Aug 13, 2021

@Nic-Ma @wyli , I also have updated the tutorial to include this handlers to trace iteration time (ITERATION_STARTED, ITERATION_COMPLETED) and batch preparation time (GET_BATCH_STARTED, GET_BATCH_COMPLETED). Project-MONAI/tutorials#310

@drbeh
Copy link
Member Author

drbeh commented Aug 16, 2021

@Nic-Ma @wyli, I'd appreciate your review here. Thanks

@drbeh drbeh added this to the Performance enhancements 0.7.0 milestone Aug 16, 2021
@drbeh drbeh mentioned this pull request Aug 16, 2021
Closed
@drbeh
Copy link
Member Author

drbeh commented Aug 16, 2021

#2728

Nic-Ma
Nic-Ma reviewed Aug 16, 2021
View reviewed changes
docs/source/handlers.rst Outdated Show resolved Hide resolved
monai/handlers/nvtx_handlers.py Outdated Show resolved Hide resolved
monai/handlers/nvtx_handlers.py Outdated Show resolved Hide resolved
monai/handlers/nvtx_handlers.py Show resolved Hide resolved
monai/handlers/nvtx_handlers.py Outdated Show resolved Hide resolved
@drbeh
Merge branch 'dev' of github.com:Project-MONAI/MONAI into nvtx-handlers
d1d8f0d
@drbeh
Update docs
3c858de 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Update event resolve method
ac3f22e 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Add test cases
5f50bb9 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Update docstring
f73b217 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Add msg input test cases
6bc2750 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Fix typing
2d8214e 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Update event setting
5242ed7 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
Nic-Ma
Nic-Ma approved these changes Aug 16, 2021
View reviewed changes
Copy link
Contributor

@Nic-Ma Nic-Ma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick update.
PR looks good to me now, @wyli could you please help also review it?

Thanks.

monai/handlers/nvtx_handlers.py Show resolved Hide resolved
wyli
wyli reviewed Aug 17, 2021
View reviewed changes
Copy link
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks it looks good to me

@drbeh
Use Events[event] for events too
84ef6ef 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Add test cases providing message for Mark and RangePush
9f78c83 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Merge branch 'dev' into nvtx-handlers
cc2684f
@drbeh drbeh enabled auto-merge (squash) August 17, 2021 13:31
@wyli wyli disabled auto-merge August 17, 2021 21:50
@wyli wyli enabled auto-merge (squash) August 17, 2021 21:50
@wyli wyli merged commit 46d5f2a into Project-MONAI:dev Aug 17, 2021
@drbeh drbeh deleted the nvtx-handlers branch August 18, 2021 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wyli wyli wyli left review comments

@Nic-Ma Nic-Ma Nic-Ma approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Performance enhancements 0.7.0
Development

Successfully merging this pull request may close these issues.

NVTX handlers to probe workflow
3 participants
@drbeh @wyli @Nic-Ma