You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Spans currently live forever on the scheduler, and with them the TaskGroups they aggregate.
This meta-data outlasts the Tasks, which are instead dereferenced as soon as they are forgotten, and will slowly increase memory usage on the scheduler.
For clusters that last a few hours, this should not be a problem, as the number of TaskGroups does not scale with the number of tasks: a workflow running on 1 megabyte of data will generate the same amount of permanent metadata as the same algorithm processing 1 petabyte.
In situations where the scheduler is never restarted, this may become a problem.
Ensure that spans are dereferenced after a certain amount of time (if and only if all of their TaskGroups are already been forgotten by the scheduler!), and with them their TaskGroups.
Note: TaskGroups are also referenced by Computations, which is deprecated functionality conceptually similar to spans, and that the Spans replaced.
The text was updated successfully, but these errors were encountered:
Spans currently live forever on the scheduler, and with them the TaskGroups they aggregate.
This meta-data outlasts the Tasks, which are instead dereferenced as soon as they are forgotten, and will slowly increase memory usage on the scheduler.
For clusters that last a few hours, this should not be a problem, as the number of TaskGroups does not scale with the number of tasks: a workflow running on 1 megabyte of data will generate the same amount of permanent metadata as the same algorithm processing 1 petabyte.
In situations where the scheduler is never restarted, this may become a problem.
Ensure that spans are dereferenced after a certain amount of time (if and only if all of their TaskGroups are already been forgotten by the scheduler!), and with them their TaskGroups.
Note: TaskGroups are also referenced by Computations, which is deprecated functionality conceptually similar to spans, and that the Spans replaced.
The text was updated successfully, but these errors were encountered: