[dashboard][train] Add dynolog for on-demand GPU profiling for Torch training

[justinvyu]

Summary

Add on-demand GPU profiling endpoint to the dashboard reporter API at /worker/gpu_profile.

Here's the list of parameters that are available:

/worker/gpu_profile?node_ip=xxx.x.x.x&pid=xxxxx&num_iterations=x

Design

• Functionality depends on dynolog binaries (dynolog, dyno) being installed on the image already.
• Launch a singleton dynolog daemon monitoring process on every node during dashboard agent startup, listening at port 65406.
• The /worker/gpu_profile request gets propagated to the ReporterAgent on the correct node, which makes a call to dyno gputrace --pids=<train_worker_pid> ... and then waits for the trace file to be dumped.
  • The request then redirects to the streaming log download API to download the trace on the client (browser).
  • See the diagram below for a visualization of the request handling.

Example E2E Usage

dynolog_prototype_e2e.mov

Edge Case Coverage

Test Case	Result
No Kineto variables set in the train run	❌ Fast-fails with an error message.
Non TorchTrainer workload	❌ Fast-fails with an error message.
On dead workers	❌ Fast-fails with an error message.
CPU workers on GPU node	⚠️ Works, but logs error: gpuGetDeviceCount failed with code 100
CPU workers on CPU nodes (no GPUs)	❌ Fast-fails with an error message.
Profiling before training steps have begun, but after workers are spawned.	❌❌ Infinite hang on the request — no .tmp file created.
Really long profile	✅ Works with 100 steps — sufficiently large test
Run ends before profiling finished (e.g. num_iterations > training duration)	✅ Returns early with a failure message if the process exits while waiting for profiling to finish.
Train V1	✅ Need to set the environment variables at the ray.init(runtime_env) level.
Multiple train runs profiled at once	✅

[justinvyu]

Will add an e2e test with real GPU + dynolog in a follow-up. For now, I've tested e2e manually.

[alanwguo]

Is this automatically hooked up to the GpuProfiling proto rpc?
Don't see any grpc code for handling it.

[justinvyu]

The proto is updated here: https://github.com/ray-project/ray/pull/53191/files#diff-1df7450350b7ffdc1923f9321c2df30c8217b6060106700e4920ba447eb7eb84

[alanwguo]

yeah, i see the proto change, but im surprised that you don't have to "hook up" the class to that proto in some way

[justinvyu]

oh yeah I think it just autogenerates the reporter stub with the new GpuProfile method