Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spans lifetime management #7977

Open
crusaderky opened this issue Jul 6, 2023 · 1 comment
Open

Spans lifetime management #7977

crusaderky opened this issue Jul 6, 2023 · 1 comment
Labels
feature Something is missing

Comments

@crusaderky
Copy link
Collaborator

crusaderky commented Jul 6, 2023

Spans currently live forever on the scheduler, and with them the TaskGroups they aggregate.
This meta-data outlasts the Tasks, which are instead dereferenced as soon as they are forgotten, and will slowly increase memory usage on the scheduler.

For clusters that last a few hours, this should not be a problem, as the number of TaskGroups does not scale with the number of tasks: a workflow running on 1 megabyte of data will generate the same amount of permanent metadata as the same algorithm processing 1 petabyte.

In situations where the scheduler is never restarted, this may become a problem.
Ensure that spans are dereferenced after a certain amount of time (if and only if all of their TaskGroups are already been forgotten by the scheduler!), and with them their TaskGroups.

Note: TaskGroups are also referenced by Computations, which is deprecated functionality conceptually similar to spans, and that the Spans replaced.

@crusaderky crusaderky added feature Something is missing and removed needs triage labels Jul 6, 2023
@fjetter
Copy link
Member

fjetter commented Jul 10, 2023

Unless they take up any significant memory, I don't consider cleaning them up very low priority

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Something is missing
Projects
None yet
Development

No branches or pull requests

2 participants