Consolidate logic to wait for dask futures.

[delucchi-cmu]

Change Description

Reducing code duplication for waiting on dask futures, updating a tqdm task bar, dask task log, raises error, and marks stage done. Also switches from explicit file names to single stage name.

Code duplication was raised as a pylint error in the SOAP PR, so I'm trying to get ahead of it.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (2802d8c) to head (23981df).
Report is 207 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #109   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           21        21           
  Lines          881       869   -12     
=========================================
- Hits           881       869   -12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[drewoldag]

This looks pretty cool. My only question here is the same as the one I posted in slack - I'm curious at what point you would make the decision to move away from dataclass to a traditional class this work.

[delucchi-cmu]

I hadn't been thinking too much about it, but now that you're asking me, I guess I will:

I'm in favor of keeping things plain-old-dataclasses for now, and for the foreseeable future. In nearly all cases, the methods that are being added are not modifying the state of the member variables, but are rather helper methods that could live just about anywhere. For the sake of code organization, it makes sense for them to live as class methods.

And I think to your question on slack

does the reduction of boilerplate that dataclass provides imply that maybe more of our traditional classes should be dataclasses

I think that's probably true, though there are certainly limitations and many things that will need to stay full-fledged classes. I have some biases that are showing (coming from years in java persistence layers, and years with protocol buffers) that make me lean toward these kind of container objects.