You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Working with a workload in a directory with lots of other workloads is much slower than it should be. This is because we're doing more build dependency checking than we really have to:
Every workload is checked for uptodate, even if it's not involved in the current command at all. This didn't used to be a big time-sync, but we now include git version checking which is significantly slower.
There is no dedup of dependency checking. In particular, almost every workload uses the built in Linux, but we check Linux for every unique workload.
We load every workload in the workload search paths every time marshal runs.
There were practical reasons for this:
it wasn't clear how to get doit to defer checking the git versions until runtime. It took some doing to get it to check at all.
Dedup would be tedious and rely on some shared state that might be confusing and lead to problems down the road.
Workloads depend on a recursive chain of 'base' options, it can be tricky to walk that dynamically while loading. It's much easier to just load everything and let doit sort it out.
The load times are the primary concern, but there are other issues as well. For example, bugs in unrelated workloads will cause a job to fail.
The solution is probably to rework the configuration loading scheme to manually trace the chain of 'base' arguments and only load workloads that actually matter. This will require a refactoring of the config loading system which might be fairly involved.
If we still want faster loads after that, we can work on deduping the dependency checking (probably by making each config_changed style criteria to its own task and performing the dedup at that point.
The text was updated successfully, but these errors were encountered:
Working with a workload in a directory with lots of other workloads is much slower than it should be. This is because we're doing more build dependency checking than we really have to:
There were practical reasons for this:
The load times are the primary concern, but there are other issues as well. For example, bugs in unrelated workloads will cause a job to fail.
The solution is probably to rework the configuration loading scheme to manually trace the chain of 'base' arguments and only load workloads that actually matter. This will require a refactoring of the config loading system which might be fairly involved.
If we still want faster loads after that, we can work on deduping the dependency checking (probably by making each config_changed style criteria to its own task and performing the dedup at that point.
The text was updated successfully, but these errors were encountered: