-
Notifications
You must be signed in to change notification settings - Fork 7k
Closed
Labels
P1Issue that should be fixed within a few weeksIssue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn'tSomething that is supposed to be working; but isn'tcommunity-backlogcore-observabilityobservabilityIssues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or ProfilingIssues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling
Description
What happened + What you expected to happen
Recently, i've run into issue where my Driver was going was running out of heap memory, and there's surprisingly no direct way to observe it on any of Ray's observability surfaces
- Dashboard doesn't list driver as the process running on the head node
- Metrics don't list driver as standalone component (though it lists dashboard, agent, etc)
This creates a gap where driver isn't really covered by any of the existing surfaces.
Versions / Dependencies
2.41
Reproduction script
Make driver script allocate and retain memory indefinitely
Try to find the source of the problem from Ray Dashboard or Metrics
Issue Severity
None
Metadata
Metadata
Assignees
Labels
P1Issue that should be fixed within a few weeksIssue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn'tSomething that is supposed to be working; but isn'tcommunity-backlogcore-observabilityobservabilityIssues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or ProfilingIssues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling