Use conda to build python packages during GPU tests

[jjacobelli]

This PR convert the from sources build we are doing in GPU test job to a conda build. This is done for the following reasons:

• This is required step to improve the Ops CI/CD setup to a more convenient pipeline
• This is required to start using conda compilers and mamba to build RAPIDS packages
• This prevent us from manually managing and installing the dependencies in GPU job
• This ensure the packages can be installed
• This ensure the tests are running and working against the package content and not the build results. Currently the Python packages are not tested.

This may increase the global pipeline time, but the usage of mamba should resolve this as mamba is faster than conda to build packages

[jjacobelli]

rerun tests

[jjacobelli]

rerun tests

[codecov-commenter]

Codecov Report

Merging #2230 (a87fe2e) into branch-22.06 (38be932) will decrease coverage by 6.66%.
The diff coverage is n/a.

@@               Coverage Diff                @@
##           branch-22.06    #2230      +/-   ##
================================================
- Coverage         70.82%   64.15%   -6.67%     
================================================
  Files               170       99      -71     
  Lines             11036     4408    -6628     
================================================
- Hits               7816     2828    -4988     
+ Misses             3220     1580    -1640     

Impacted Files	Coverage Δ
python/cugraph/cugraph/__init__.py	100.00% <ø> (ø)
python/cugraph/cugraph/centrality/__init__.py	100.00% <ø> (ø)
...graph/cugraph/centrality/betweenness_centrality.py	89.65% <ø> (ø)
...thon/cugraph/cugraph/centrality/katz_centrality.py	88.23% <ø> (-1.24%)	⬇️
python/cugraph/cugraph/community/egonet.py	97.36% <ø> (ø)
...ython/cugraph/cugraph/community/ktruss_subgraph.py	85.29% <ø> (ø)
...cugraph/cugraph/dask/centrality/katz_centrality.py	25.00% <ø> (ø)
python/cugraph/cugraph/dask/common/input_utils.py	22.13% <ø> (+0.57%)	⬆️
python/cugraph/cugraph/dask/common/mg_utils.py	30.43% <ø> (-7.07%)	⬇️
python/cugraph/cugraph/dask/common/part_utils.py	19.23% <ø> (-0.94%)	⬇️
... and 121 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 081df0a...a87fe2e. Read the comment docs.

[jjacobelli]

rerun tests

[ajschmidt8]

Approving ops-codeowner file changes.

[rlratzel]

Most of the changes look okay, but I did see a few that could cause problems which I commented on below. I also don't understand why some of these changes are needed for allowing the GPU tests to use a conda install instead of a from-source build (eg. why can't we continue to have a cugraph.tests package? Why do we need a separate cugraph.testing package?, etc.). Maybe we can chat offline about it and then update the PR or report back here, if necessary?

[rlratzel]

A few things:

• this does not add the tests directory to the import path, it adds .., which will be interpreted as the parent directory of the CWD from where pytest is run.
• pytest automatically adds the directories containing the collected test modules to the import path, so this may be "accidentally" working because of that
• these changes are not being tested by CI since our test script skips everything in the mg directory, so have you manually tested these changes on a multi-GPU machine to ensure they work?
• needing to modify sys.path directly could be a symptom that we have a broken package hierarchy or some other code smell - is there a reason this directory is no longer a package (ie. had its __init__.py file removed)?

[rlratzel]

@Ethyling based on our conversation, would you mind removing the sys.path.insert() calls and adding a FIXME stating that those tests need to be refactored to not import other tests and should instead put shared test code in a util module? Same for the other comment below.

[rlratzel]

(same feedback as above)

[rlratzel]

@Ethyling and I just had a call about this and we agreed on the following:

• The changes to remove the __init__.py file so cugraph.tests is no longer a package is to force pytests to find imported modules from installs rather than relative to the source tree where the tests are located. pytest by default will walk the test location tree upwards until it doesn't find a package (a directory with an __init__.py file), and then use that as the base dir for imports, as documented here. This is nice for dev testing since it means devs don't have to install each time they change code, but causes problems here since the source tree no longer contains builds of python/C extensions (because now they're installed from conda). There is an option to tell pytest to add the source tree to the end of the import path - which should resolve this - but other teams like cuML (cc @dantegd ) preferred to just remove the test __init__.py file. Since it's good to be consistent with other teams, and because our tests at the moment may not run outside of a dev environment (they have additional deps not listed in our conda recipe) and therefore shouldn't be an importable package, removing cugraph.tests as a package makes sense. We may revisit this later if we realize there's advantages to keeping cugraph.tests as an importable package.
• The cugraph.testing package was added to make it easier to do absolute imports from tests of common utilities, since cugraph.tests is no longer a package. This is also something done by other teams such as cuML and cuDF.
• The sys.path.insert(...) calls are still a problem, however, since these are only used by MG tests, and MG tests are not run as part of CI, these won't break PRs. @Ethyling mentioned he could add a FIXME to these tests since we agreed it would be better if they could be refactored to not have tests importing other tests anyway (instead, have the shared parts refactored into a common file, possibly in cugraph.testing).

[jjacobelli]

@gpucibot merge