[BUG] CI test time increased by 1/2 hour during 0.16 release cycle

[JohnZed]

Timings are consistently a bit over 2 hours where they were previously 1.5 hours.

See the change around pipeline 96 here: https://gpuci.gpuopenanalytics.com/blue/organizations/jenkins/rapidsai%2Fgpuci%2Fcuml%2Fbranches%2Fcuml-cpu-branch-0.16/activity/

Timings are variable, but on one A100 example they look like roughly:

• 20 min build from source
• 4-5 min C++ and prims tests
• 40 min single-gpu tests
• 40 min Dask tests
• 3 min notebook tests
• 3-4 min env setup and miscellaneous

[JohnZed]

TODO:

• Expanded timing of build steps (not just pytest - @JohnZed )
• Detailed pytest breakdown (@dantegd)
• Look into how to improve Jenkins build time report (@JohnZed to look in)
• Look with ops into options of optional lighter-weight test runs (e.g. just run a single target)

[github-actions]

This issue has been marked rotten due to no recent activity in the past 90d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.