Skip to content

Conversation

@wchargin
Copy link
Contributor

@wchargin wchargin commented Feb 6, 2021

Summary:
The DebuggerV2 plugin reads data outside of the usual summary-based
workflows, and has its own loading infrastructure for doing so. Until
this patch, the plugin would start recursively scanning the user’s log
directory at plugin initialization time—even before the server launches.
This can take a lot of resources and introduce a lot of contention,
especially on remote file systems.

This patch defers all such logic until users of the frontend explicitly
activate the plugin. Startup time improves greatly (see below).

Test Plan:
When pointing at gs://tensorboard-bench-logs, the time between process
start and the “server ready” line drops from 9.5 seconds to 1.7 seconds,
as measured by:

bazel-bin/tensorboard/tensorboard --logdir gs://tensorboard-bench-logs 2>&1 \
    | ts -s '[%.s]'

After generating test data as described in the tutorial, the plugin is
indeed inactive by default, and does indeed still appear to work if
manually activated.

wchargin-branch: dbgv2-lazy-load

Summary:
The DebuggerV2 plugin reads data outside of the usual summary-based
workflows, and has its own loading infrastructure for doing so. Until
this patch, the plugin would start recursively scanning the user’s log
directory at plugin initialization time—even before the server launches.
This can take a lot of resources and introduce a lot of contention,
especially on remote file systems.

This patch defers all such logic until users of the frontend explicitly
activate the plugin. Startup time improves greatly (see below).

Test Plan:
When pointing at `gs://tensorboard-bench-logs`, the time between process
start and the “server ready” line drops from 9.5 seconds to 2.6 seconds,
as measured by:

```
bazel-bin/tensorboard/tensorboard --logdir gs://tensorboard-bench-logs 2>&1 \
    | ts -s '[%.s]'
```

After generating test data as described in the tutorial, the plugin is
indeed inactive by default, and does indeed still appear to work if
manually activated.

wchargin-branch: dbgv2-lazy-load
wchargin-source: 1331fc5eec5668090967d335bbda48960ed9a9e6
@wchargin wchargin added type:bug plugin:debugger theme:performance Performance, scalability, large data sizes, slowness, etc. labels Feb 6, 2021
@google-cla google-cla bot added the cla: yes label Feb 6, 2021
@wchargin wchargin requested a review from nfelt February 6, 2021 02:15
wchargin-branch: dbgv2-lazy-load
wchargin-source: 1fa1a8768ad751b55bf4919a00d6b3909b04ca57
@wchargin wchargin merged commit a8a1e18 into master Feb 9, 2021
@wchargin wchargin deleted the wchargin-dbgv2-lazy-load branch February 9, 2021 21:36
@wchargin wchargin mentioned this pull request Feb 16, 2021
34 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: yes plugin:debugger theme:performance Performance, scalability, large data sizes, slowness, etc. type:bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants