Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Selective Workload Loading #149

Open
NathanTP opened this issue May 4, 2020 · 1 comment
Open

Selective Workload Loading #149

NathanTP opened this issue May 4, 2020 · 1 comment

Comments

@NathanTP
Copy link
Contributor

NathanTP commented May 4, 2020

Working with a workload in a directory with lots of other workloads is much slower than it should be. This is because we're doing more build dependency checking than we really have to:

  1. Every workload is checked for uptodate, even if it's not involved in the current command at all. This didn't used to be a big time-sync, but we now include git version checking which is significantly slower.
  2. There is no dedup of dependency checking. In particular, almost every workload uses the built in Linux, but we check Linux for every unique workload.
  3. We load every workload in the workload search paths every time marshal runs.

There were practical reasons for this:

  1. it wasn't clear how to get doit to defer checking the git versions until runtime. It took some doing to get it to check at all.
  2. Dedup would be tedious and rely on some shared state that might be confusing and lead to problems down the road.
  3. Workloads depend on a recursive chain of 'base' options, it can be tricky to walk that dynamically while loading. It's much easier to just load everything and let doit sort it out.

The load times are the primary concern, but there are other issues as well. For example, bugs in unrelated workloads will cause a job to fail.

The solution is probably to rework the configuration loading scheme to manually trace the chain of 'base' arguments and only load workloads that actually matter. This will require a refactoring of the config loading system which might be fairly involved.

If we still want faster loads after that, we can work on deduping the dependency checking (probably by making each config_changed style criteria to its own task and performing the dedup at that point.

@NathanTP
Copy link
Contributor Author

Partially solved by #159 which greatly improves the performance of dependency checking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant