tb_plugin: fix startup hang when no profile data #326
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When starting up tensorboard and the plugin in a directory where there is no profile data, `TorchProfilerPlugin.is_active()` hangs forever. This makes the tensorboard server hang, and client-side webpage remain blank. This commit fixes the issue by ensuring that the thread monitoring the run directories for data notifies the `_is_active_initialized_event` flag not only when data is found, but also when the search is complete and no data was found. A simple unit test is added which validates the fix: without this commit, the test hangs indefinitely.
mautier
force-pushed
the
bugfix/tb_plugin/fix_startup_hang_when_no_data
branch
from
8740b99 to
a4e3406
Compare
Contributor
|
This is a known issue, #270. We will think about it and gave a symmetrical fix. |
Contributor
|
The fix is no exhaustive and it will be closed. |
Contributor
|
#340 should fix the issue |
Author
|
Awesome thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When starting up tensorboard and the plugin in a directory where there
is no profile data,
TorchProfilerPlugin.is_active()hangs forever. Thismakes the tensorboard server hang, and client-side webpage remain blank.
This commit fixes the issue by ensuring that the thread monitoring the
run directories for data notifies the
_is_active_initialized_eventflag not only when data is found, but also when the search is complete
and no data was found.
A simple unit test is added which validates the fix: without this
commit, the test hangs indefinitely.