Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update_branch #4

Open
wants to merge 442 commits into
base: main
Choose a base branch
from
Open

update_branch #4

wants to merge 442 commits into from

Conversation

jnke2016
Copy link
Owner

update main branch

rlratzel and others added 30 commits January 21, 2021 16:18
Implement induced subgraph extraction (SG C++)
Updated SOURCEBUILD.md to include the latest build.sh options
This PR prepares the changelog to be automatically updated during releases.

Authors:
  - AJ Schmidt (@ajschmidt8)

Approvers:
  - Rick Ratzel (@rlratzel)

URL: rapidsai/cuml#1368
Close #1313 #1331 #1330

Authors:
  - @Iroy30

Approvers:
  - Brad Rees (@BradReesWork)
  - Rick Ratzel (@rlratzel)
  - Alex Fender (@afender)

URL: #1332
This PR adds the GitHub action [PR Labeler](https://github.com/actions/labeler) to auto-label PRs based on their content. 

Labeling is managed with a configuration file `.github/labeler.yml` using the following [options](https://github.com/actions/labeler#usage).

Authors:
  - Joseph (@jolorunyomi)

Approvers:
  - AJ Schmidt (@ajschmidt8)
  - Rick Ratzel (@rlratzel)

URL: #1358
added the coverage plugin option by default
added pytest-cov package

Authors:
  - @jnke2016

Approvers:
  - AJ Schmidt (@ajschmidt8)
  - Brad Rees (@BradReesWork)

URL: #1352
Bug fixes for MNMG coarsen_graph, renumber_edgelist, relabel

Authors:
  - Seunghwa Kang (@seunghwak)

Approvers:
  - Andrei Schaffer (@aschaffer)
  - Rick Ratzel (@rlratzel)
  - Chuck Hastings (@ChuckHastings)
  - Alex Fender (@afender)

URL: #1364
updated the conda environments to have cmake 3.18 and update the README file

Authors:
  - Brad Rees (@BradReesWork)

Approvers:
  - AJ Schmidt (@ajschmidt8)
  - Rick Ratzel (@rlratzel)
  - Alex Fender (@afender)

URL: #1369
This exposes to cython / python newly added C++ functions `sort_and_shuffle_values()` and `renumber_edgelist()` in order to perform MNMG renumbering.

Authors:
  - Andrei Schaffer (@aschaffer)

Approvers:
  - Chuck Hastings (@ChuckHastings)
  - Seunghwa Kang (@seunghwak)
  - Alex Fender (@afender)

URL: #1355
…t cugraph as an external project and other tech debt removal (#1367)

This PR makes cuGraph's cmake files more consistent with other RAPIDS libs by matching the minimum required cmake version, adding `project()` statements to cugraph's thirdparty modules, and using `CMAKE_CURRENT_SOURCE_DIR` appropriately so paths are relative to the CMakeLists.txt file rather than the top-level cmake dir of the project (since that may not be the cugraph cpp dir in the case of cugraph being used as an external project by another application).

This also adds a `CUDA_ARCHITECTURES=OFF` setting to suppress the warning printed for each test target. This setting may be replaced/changed once the findcudatoolkit feature is used in a future cmake version.

This also removes the Arrow and GTest cmake files since Arrow is not a direct dependency and those files were not being used, and GTest is now a build requirement in the conda dev environment and does not need to be built from source (the conda dev env files have been updated accordingly).

This PR also addresses much of #1075 , but not completely since gunrock is still using `ExternalProject` due to (I think) updates that need to be made to their cmake files to support this.

This was tested by observing a successful clean build, however it was **not** tested by creating a separate cmake application to simulate cugraph being used as a 3rd party package.

Note: the changes in this PR were modeled after rapidsai/rmm#541

closes #1137 
closes #1266

Authors:
  - Rick Ratzel (@rlratzel)

Approvers:
  - Chuck Hastings (@ChuckHastings)
  - AJ Schmidt (@ajschmidt8)
  - Brad Rees (@BradReesWork)

URL: #1367
### Description
Let the egonet graph of a node x be the subgraph that includes the neighborhood of x and all edges between them. Here is a basic description (1-hop, single seed) :
- Add center node x to the graph.
- Go through all the neighbors y of this center node x, add edge (x, y) to the graph. 
- For each neighbor y of center node x, go through all the neighbors z of center node x, if there is an edge between y and z in the original graph, add edge (y, z) to our new graph. 

### Proposed solution
Rather than doing custom one/two hops features, we propose a generic k-hops solution leveraging BFS with cutoff to identify neighbors within a given radius. 

In addition to the single source version (matching what's available in Nx), we propose to handle multiple sources (seeds) at once which allows better performances.

This PR also enables a path in the experimental stack for returning multiple graphs (edge list format) from CUDA prims to python without using the legacy classes.

As future work, we want to enable concurrency for the cutoff BFS for each seed. This is dependent of #957 

Close #475

Authors:
  - Alex Fender (@afender)
  - @Iroy30

Approvers:
  - @Iroy30
  - Brad Rees (@BradReesWork)

URL: #1365
Authors:
  - @Iroy30

Approvers:
  - Brad Rees (@BradReesWork)
  - Alex Fender (@afender)

URL: #1378
Preparing for MNMG Leiden and ECG identified an area for code cleanup.

The original cuGraph implementation of Louvain would flatten the hierarchical clustering as it was computed, filling (and returning) the final clustering.  This adds an awkward step in the middle of the Louvain computation.  Additionally, since Louvain (and Leiden and ECG which derive from it) is actually a hierarchical clustering algorithm it would be nice to generate the actual Dendrogram.

This PR implements a Dendrogram class, a function for flattening the Dendrogram, and modifies Louvain, Leiden and ECG to use the Dendrogram class.

It was suggested that the Dendrogram class could be moved to raft, decided to defer that until later, it's easy enough to move.

Authors:
  - Chuck Hastings (@ChuckHastings)

Approvers:
  - Alex Fender (@afender)
  - Brad Rees (@BradReesWork)
  - Andrei Schaffer (@aschaffer)
  - Rick Ratzel (@rlratzel)
  - AJ Schmidt (@ajschmidt8)

URL: #1359
Fix forward-merge conflicts for #1370 [skip-ci]
…ted LD_LIBRARY_PATH for project flash runs (#1386)

* Updated CI scripts to use a different error handling convention
* Updated LD_LIBRARY_PATH for project flash runs
* Added extra logging to report status after each test command
* Added comments
* Removed unused "top-20 slowest" report
* Minor updates for consistency

Tested locally by simulating various error conditions (removed .so files, inserted errors in NBs, killed processes, etc.) and checked exit codes.  Still need to verify in a project Flash env, but using CI for that.

Authors:
  - Rick Ratzel (@rlratzel)

Approvers:
  - AJ Schmidt (@ajschmidt8)
  - Ray Douglass (@raydouglass)
  - Dillon Cullinan (@dillon-cullinan)

URL: #1386
[gpuCI] Auto-merge branch-0.18 to branch-0.19 [skip ci]
Modify experimental::louvain to use the new `coarsen_graph` primitive.  This replaces the original implementation of `shrink_graph`.

Authors:
  - Chuck Hastings (@ChuckHastings)

Approvers:
  - Andrei Schaffer (@aschaffer)
  - Rick Ratzel (@rlratzel)
  - Seunghwa Kang (@seunghwak)

URL: #1362
[gpuCI] Auto-merge branch-0.18 to branch-0.19 [skip ci]
…t using it (#1361)

Added initial infrastructure for MG C++ testing and a Pagerank MG test using it.

<s>Still a WIP, need to:</s>
* <s>Shuffle step is currently failing</s>
* <s>`graph_t` ctor expensive check is failing</s>
* <s>Finish comparison code to reference SG Pagerank results</s>
* <s>Fix the `#include` guard hack in `test_utilities.hpp`</s>
* <s>Lots of cleanup</s>
* <s>Refactor common steps into proper `SetUp()` and `TearDown()` functions</s>

closes #1136

Authors:
  - Rick Ratzel (@rlratzel)
  - Seunghwa Kang (@seunghwak)

Approvers:
  - Brad Rees (@BradReesWork)
  - Andrei Schaffer (@aschaffer)
  - Chuck Hastings (@ChuckHastings)

URL: #1361
seunghwak and others added 28 commits July 9, 2021 13:05
Louvain currently crashes with 64 bit vertex ID (thrust::copy with thrust::make_transform_iterator and a device lambda returns cudaErrorInvalidDeviceFunction). This PR fixes this error (replaces the device lambda with a ```struct minor_to_key_t``` functor).

Authors:
  - Seunghwa Kang (https://github.com/seunghwak)

Approvers:
  - Chuck Hastings (https://github.com/ChuckHastings)

URL: #1696
…etting UCX install location (#1698)

This PR is a continuation of #1694. Similar to rapidsai/cuml#4015, this PR updates setup.py to:

* Use `library_dirs` instead of `runtime_library_dirs` when linking Cython.
* Allow overriding UCX lib and include dirs via a `UCX_HOME` envvar.
* Link `cudart`, `cusparse`, and `cusolver`.

These are necessary to compile the Cython via `pip` when not inside a conda environment and when UCX is installed to a location other than `/usr` or `/usr/local`.

Authors:
  - Paul Taylor (https://github.com/trxcllnt)

Approvers:
  - Rick Ratzel (https://github.com/rlratzel)

URL: #1698
any_of_adj_matrix_row & transform_reduce_v_with_adj_matrix_row (graph primitives) were initially added to be used in error checking but they are no longer used (and may not be used in the future as well).

Deleting the dead code.

Authors:
  - Seunghwa Kang (https://github.com/seunghwak)

Approvers:
  - Chuck Hastings (https://github.com/ChuckHastings)

URL: #1704
Update graph primitives to support DCSR (DCSC) segments (except for the ones used by Louvain, graph primitives used in Louvain will be updated in a separate PR with thread-divergence optimization & more testing).

DCSR (DCSC) segment support is still disabled (as enabling this will break Louvain).

Authors:
  - Seunghwa Kang (https://github.com/seunghwak)
  - Mark Harris (https://github.com/harrism)
  - Chuck Hastings (https://github.com/ChuckHastings)
  - AJ Schmidt (https://github.com/ajschmidt8)

Approvers:
  - Chuck Hastings (https://github.com/ChuckHastings)
  - Brad Rees (https://github.com/BradReesWork)

URL: #1690
replace `cudf assert_eq` by `assert_frame_equal` which is more stable

Authors:
  - Joseph Nke (https://github.com/jnke2016)

Approvers:
  - Brad Rees (https://github.com/BradReesWork)

URL: #1693
Updated cugraph recipe to include CUDA version in build string and depend on a specific cudatoolkit based on CUDA version. This pattern was copied from cuML's recipe.

closes #1708

Authors:
  - Rick Ratzel (https://github.com/rlratzel)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)

URL: #1709
Added test for count_if primitive.

Authors:
  - Kumar Aatish (https://github.com/kaatish)

Approvers:
  - Seunghwa Kang (https://github.com/seunghwak)
  - Chuck Hastings (https://github.com/ChuckHastings)

URL: #1697
This PR removes some conditional statements & variables that prevent all the variants of the `cugraph` package from being uploaded to Anaconda.org. This is a continuation of #1709.

Authors:
  - AJ Schmidt (https://github.com/ajschmidt8)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)
  - Jordan Jacobelli (https://github.com/Ethyling)
  - Brad Rees (https://github.com/BradReesWork)

URL: #1712
Added generator to API docs
Removed reference to 11.2 environment
Updated list of algorithms

Authors:
  - Brad Rees (https://github.com/BradReesWork)

Approvers:
  - Chuck Hastings (https://github.com/ChuckHastings)

URL: #1719
- [x] Update key-based primitives (used by Louvain) to reduce thread-divergence
- [x] Update key-based primitives (used by Louvain) to support DCSR (DCSC) segments

Authors:
  - Seunghwa Kang (https://github.com/seunghwak)

Approvers:
  - Chuck Hastings (https://github.com/ChuckHastings)
  - Kumar Aatish (https://github.com/kaatish)

URL: #1703
This enhancement PR provides the C++ infrastructure for a type-erased API to be exposed to C-only, Cython, and potentially other glue languages. This will be the first in a series of such API PRs to expose cugraph functionality to other languages.

Authors:
  - Andrei Schaffer (https://github.com/aschaffer)
  - Mark Harris (https://github.com/harrism)
  - AJ Schmidt (https://github.com/ajschmidt8)

Approvers:
  - Chuck Hastings (https://github.com/ChuckHastings)
  - Kumar Aatish (https://github.com/kaatish)

URL: #1711
We had an issue in CI that uncovered the fact that raft is referencing the `dev` branch of coco while `cugraph` has it pinned to a previous commit.  CI is using the newer `dev` branch version, which doesn't match what happens when we do local builds.

The original motivation for pinning was a temporary fix because of a bug introduced in `coco` which has since been resolved.  This PR reverts us back to using the `dev` branch.

Authors:
  - Chuck Hastings (https://github.com/ChuckHastings)

Approvers:
  - Brad Rees (https://github.com/BradReesWork)

URL: #1721
Some changes to speed up compilation of cugraph.  With 8 CPUs, this speeds up the overall compile time on my workstation from 13 minutes to about 9 minutes.  Should speed up C++ compilation in CI by about 10 minutes total.

Authors:
  - Chuck Hastings (https://github.com/ChuckHastings)

Approvers:
  - Brad Rees (https://github.com/BradReesWork)
  - Andrei Schaffer (https://github.com/aschaffer)
  - Kumar Aatish (https://github.com/kaatish)

URL: #1702
Fix `test_mg_batch_betweenness_centrality`, `test_mg_batch_edge_betweenness_centrality`, `test_mg_replication`, `test_mg_renumber`

closes #1706 
closes #1715 
closes #1716

Authors:
  - Joseph Nke (https://github.com/jnke2016)

Approvers:
  - Rick Ratzel (https://github.com/rlratzel)
  - Brad Rees (https://github.com/BradReesWork)

URL: #1718
Added test for reduce_v primitive.

Authors:
  - Kumar Aatish (https://github.com/kaatish)

Approvers:
  - Seunghwa Kang (https://github.com/seunghwak)
  - Chuck Hastings (https://github.com/ChuckHastings)

URL: #1710
Using the latest `cuco` commit hash as the `GIT_TAG` for release purposes.  See also RAFT PR rapidsai/raft#304 .
Also updated a debug message.

NOTE: using `skip-ci` until rapidsai/raft#304 is merged.
NOTE: The commit hash will likely be reverted to `dev` in 21.10 during development, then a new hash will be used for 21.10 release, and so on.

Authors:
  - Rick Ratzel (https://github.com/rlratzel)

Approvers:
  - Chuck Hastings (https://github.com/ChuckHastings)

URL: #1741
Fixed indentations so that docs appear correctly

Authors:
  - Brad Rees (https://github.com/BradReesWork)

Approvers:
  - Rick Ratzel (https://github.com/rlratzel)

URL: #1735
…added as a target (#1746)

Changed cuco cmake function to return early if cuco has already been added as a target.  This matches the technique used by raft [here](https://github.com/rapidsai/raft/blob/a3af3895410c19f3e713caa608ea2024f6008350/cpp/cmake/thirdparty/get_cuco.cmake#L19).

Tested by doing a build and install of cuML, followed by a build of cuGraph and observing a CPM error about the alias target `cuco::cuco` already existing.  Made the change to return early if cuco is already a target and observed the cuGraph `libcugraph.so` build succeed.

Authors:
  - Rick Ratzel (https://github.com/rlratzel)

Approvers:
  - Chuck Hastings (https://github.com/ChuckHastings)
  - Dillon Cullinan (https://github.com/dillon-cullinan)

URL: #1746
Co-authored-by: ptaylor <paul.e.taylor@me.com>
…1773)

Added guards for preventing k-truss execution on CUDA 11.4 systems and the appropriate user-friendly messaging, along with a new unit test.
Also updated the k-truss notebook with a note about the unsupported CUDA version and a comment to skip automated tests on CUDA 11.4, along with updates to the notebook skip mechanism to also look at CUDA version.

Tested on a CUDA 11.2 system by changing the unsupported version to 11.2 to verify correct behavior.

Behavior on a CUDA 11.4 systems should be to allow cugraph to be imported as always, but if the user calls k_truss or kturss_subgraph, a NotImplementedError exception with "is not currently supported in CUDA 11.4 environments" message is raised. If the user directly imports the module (ie. from cugraph.community.ktruss_subgraph import k_truss), the exception is raised immediately.

Behavior on non-CUDA 11.4 systems should be exactly the same as prior to this change.
Updated to handle CUDA version check on machines with no GPUs. When no GPUs are present, cugraph can import and ktruss is available, which should match the original behavior prior to when the checks were added for disabling ktruss on CUDA 11.4

Tested by setting CUDA_VISIBLE_DEVICES to " ", verifying the expected exception was raised, made the change, then verified cugraph could be imported.
Removes `-g` from the compile commands generated by distutils to compile Cython files.

This will make our container images, conda packages, and python wheels smaller.
@github-actions
Copy link

github-actions bot commented Nov 4, 2021

This PR has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this PR if it is no longer required. Otherwise, please respond with a comment indicating any updates. This PR will be labeled inactive-90d if there is no activity in the next 60 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.