aschaffer/21.08 <- rapidsai/21.08 #48
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Bindings for RMAT Bindings for the list of RMAT graphs includes some quick fixes to address some sphinx doc-building warnings closes #1473 Authors: - Joseph Nke (https://github.com/jnke2016) - Rick Ratzel (https://github.com/rlratzel) Approvers: - Andrei Schaffer (https://github.com/aschaffer) - Chuck Hastings (https://github.com/ChuckHastings) - Rick Ratzel (https://github.com/rlratzel) URL: #1573
in MNMG mode, clusters assigned to different GPUs can increase between iterations. Merge after #1596 closes #1586 Authors: - Chuck Hastings (https://github.com/ChuckHastings) Approvers: - Brad Rees (https://github.com/BradReesWork) URL: #1597
- [x] Bug fix in MG coarsen_graph Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) URL: #1596
This PR moves the headers in `cpp/include` into `cpp/include/cugraph` and updates C++ and Cython `#include` directives. This change makes it easier for other libs to avoid naming conflicts with cuGraph public headers when linking `libcugraph.so`. closes #1491 Authors: - Paul Taylor (https://github.com/trxcllnt) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) - Seunghwa Kang (https://github.com/seunghwak) - Rick Ratzel (https://github.com/rlratzel) URL: #1582
Fix: - Broken path utility Authors: - Brad Rees (https://github.com/BradReesWork) Approvers: - Rick Ratzel (https://github.com/rlratzel) URL: #1576
Relabel function relabels old labels to new labels based on the provided (old_label, new_label) pairs. The provided pairs should include the entire set of old labels to be relabeled. This updates add an additional parameter ```skip_missing_labels``` to skip relabeling for labels missing in the provided pairs. This update is necessary for the new WCC implementation. Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) URL: #1598
Update the release script to take a parameter with the new version instead of calculating the new version. Authors: - Ray Douglass (https://github.com/raydouglass) Approvers: - Dillon Cullinan (https://github.com/dillon-cullinan) - AJ Schmidt (https://github.com/ajschmidt8) URL: #1600
… a prior build/install step (#1601) Added `uninstall` target which uninstalls libcugraph and cugraph from a prior build/install step. Also extended the `clean` target to remove inplace artifacts (mainly cython-generated cpp files and inplace `.so` python extensions built from cython). NOTE: since targets can be combined on the same command, a user can get the "scrub" behavior by running: ``` ./build.sh uninstall clean ``` Authors: - Rick Ratzel (https://github.com/rlratzel) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) - Brad Rees (https://github.com/BradReesWork) URL: #1601
…on (#1551) To get early feedback and to create a baseline weakly-connected-component PR dependent on this PR. This PR requires C++17 and will not compile till #1528 gets resolved. Tagging vertex IDs with root IDs is necessary in multi-source BFS, k-hop neighbors, and the newly designed weakly-connected-component algorithm based on multi-root collaborative frontier expansion. To fully support multi-source BFS & k-hop neighbors, we also need to store/query properties for (vertex, tag) pairs, and this update will be added in a future PR. Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) URL: #1551
- Move the graph generation function from ```tests/utilities``` to ```src/generator```. - Use std::optional to take the optional vertex list parameter instead of using two separate functions. Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) - Andrei Schaffer (https://github.com/aschaffer) URL: #1565
…mmunity to tests/utilities and update MG tests (#1602) - Move the utility function to gather distributed vectors from tests/community/mg_louvain_helper.cu to tests/utilities/device_comm_wrapper.cu and rename the function to device_gatherv - Update MG tests to gather the MG results in root and compare with the SG result in root. - Few more minor updates (sort_by_key thrust wrapper return value, adding missing const in input parameter, add missing include statement, add barrier in MG performance measurement) This update is necessary to simplify MG WCC testing. Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) URL: #1602
This PR removes a variable that is no longer necessary for docs builds after the calver transition. Authors: - AJ Schmidt (https://github.com/ajschmidt8) Approvers: - Dillon Cullinan (https://github.com/dillon-cullinan) URL: #1606
- C++ SG & MG Weakly Connected Components - This PR is to help python binding development - Performance/memory footprint optimizations for SG, MG with # GPUs < (E/V)^2, MG with # GPUs > (E/V)^2 will be in separate PRs Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Andrei Schaffer (https://github.com/aschaffer) - Chuck Hastings (https://github.com/ChuckHastings) URL: #1604
This PR updates the `0.20` references in `CHANGELOG.md` to be `21.06`. Authors: - AJ Schmidt (https://github.com/ajschmidt8) Approvers: - Dillon Cullinan (https://github.com/dillon-cullinan) - Brad Rees (https://github.com/BradReesWork) URL: #1608
- Performance tune/reduce memory footprint for WCC for SG & MG (when # GPUs < (E/V)^2, additional memory footprint optimizations are necessary if # GPUs > (E/V)^2 and this will be addressed in separate PRs). - MG bug fixes Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Andrei Schaffer (https://github.com/aschaffer) - Chuck Hastings (https://github.com/ChuckHastings) URL: #1605
RMM is going to require specifying the stream in the device_buffer constructor. This changes the one `cugraph` item and it updates to a version of `cuhornet` that makes the necessary changes. Authors: - Chuck Hastings (https://github.com/ChuckHastings) Approvers: - Seunghwa Kang (https://github.com/seunghwak) URL: #1609
Fix merge conflicts [skip ci]
…endencies (#1585) This allows the `cugraph` to be used via `CPM` either from a system installed version or from a build-directory. Authors: - Robert Maynard (https://github.com/robertmaynard) - Paul Taylor (https://github.com/trxcllnt) - Dante Gama Dessavre (https://github.com/dantegd) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) - Rick Ratzel (https://github.com/rlratzel) URL: #1585
[gpuCI] Forward-merge branch-21.06 to branch-21.08 [skip ci]
After recent RMM changes, the DEVELOPER_GUIDE.md examples for `rmm::device_buffer` needed to be updated/corrected. Authors: - Mark Harris (https://github.com/harrism) Approvers: - Christopher Harris (https://github.com/cwharris) - Chuck Hastings (https://github.com/ChuckHastings) URL: #1619
[gpuCI] Forward-merge branch-21.06 to branch-21.08 [skip ci]
#1610) `G.from_cudf_edgelist` after the `cudf.read_csv` fails with the following error - ``` --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-7-d38ba590f728> in <module> 1 t_start_build_sg = time.perf_counter() 2 G = cugraph.DiGraph() ----> 3 G.from_cudf_edgelist(e_list, source='src', destination='dst') 4 t_stop_build_sg = time.perf_counter() /opt/conda/envs/rapids/lib/python3.7/site-packages/cugraph/structure/graph_classes.py in from_cudf_edgelist(self, input_df, source, destination, edge_attr, renumber) 127 destination=destination, 128 edge_attr=edge_attr, --> 129 renumber=renumber) 130 131 def from_cudf_adjlist(self, offset_col, index_col, value_col=None): /opt/conda/envs/rapids/lib/python3.7/site-packages/cugraph/structure/graph_implementation/simpleGraph.py in __from_edgelist(self, input_df, source, destination, edge_attr, renumber) 171 source_col, dest_col = symmetrize( 172 source_col, dest_col, multi=self.properties.multi_edge, --> 173 symmetrize=not self.properties.directed) 174 175 self.edgelist = simpleGraphImpl.EdgeList(source_col, dest_col, /opt/conda/envs/rapids/lib/python3.7/site-packages/cugraph/structure/symmetrize.py in symmetrize(source_col, dest_col, value_col, multi, symmetrize) 200 ) 201 csg.null_check(source_col) --> 202 csg.null_check(dest_col) 203 if value_col is not None: 204 if isinstance(value_col, cudf.Series): /opt/conda/envs/rapids/lib/python3.7/site-packages/cugraph/structure/graph_classes.py in null_check(col) 23 def null_check(col): 24 if col.null_count != 0: ---> 25 raise ValueError("Series contains NULL values") 26 27 ValueError: Series contains NULL values ``` the header from the input is turning into NULLs - ``` $ head soc-LiveJournal1.txt # Directed graph (each unordered pair of nodes is saved once): soc-LiveJournal1.txt # Directed LiveJournal friednship social network # Nodes: 4847571 Edges: 68993773 # FromNodeId ToNodeId 0 1 0 2 0 3 ``` `e_list` ||src|dst |---|---|--- |0|0|NA |1|0|NA |2|0|NA |3|0|0 |4|0|1 ... this issue has existed since at least 0.16 and may warrant further test coverage for cudf's read_csv. Authors: - Matthew Farrellee (https://github.com/mattf) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) - Brad Rees (https://github.com/BradReesWork) URL: #1610
[gpuCI] Forward-merge branch-21.06 to branch-21.08 [skip ci]
Disabling MG C++ testing outputs for non-root processes has an undesirable side effect of disabling error messages (e.g. exception outputs) as well. This makes failure diagnosis difficult in large-scale runs. Delete the code disabling MG C++ testing outputs for non-root processes in this PR. In long run, we need a more elegant mechanism to disable only redundant test progress messages while still allowing error messages to be displayed. Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) URL: #1615
Currently cuco::static_map instances in cuGraph are initialized with the cuco::static_map default allocator (which uses CUDA memory allocator). Update to provide the RMM memory allocator instead. Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) URL: #1617
[gpuCI] Forward-merge branch-21.06 to branch-21.08 [skip ci]
- Update wrapper to handle int64 - Use `device_uvector` Close: #1581 Authors: - Hugo Linsenmaier (https://github.com/hlinsen) Approvers: - Andrei Schaffer (https://github.com/aschaffer) - Chuck Hastings (https://github.com/ChuckHastings) - Brad Rees (https://github.com/BradReesWork) URL: #1607
[gpuCI] Forward-merge branch-21.06 to branch-21.08 [skip ci]
Updated dependencies for CalVer Note: I have not tested this yet. Authors: - Rick Ratzel (https://github.com/rlratzel) Approvers: - Brad Rees (https://github.com/BradReesWork) - AJ Schmidt (https://github.com/ajschmidt8) URL: #1629
…ec_policy` (#1625) Fixes an issue where the layout for all points converge to `(0,0)` when running multiple layout ticks after #1607. * Initializes the `d_old_forces` vector to fix layout issue. * Updates to `barnes_hut.hpp` to use non-deprecated `rmm::exec_policy`. Authors: - Paul Taylor (https://github.com/trxcllnt) Approvers: - Hugo Linsenmaier (https://github.com/hlinsen) - Alex Fender (https://github.com/afender) URL: #1625
This tracks work on graph object serialization and broadcasting functionality: 1. C++ support of un/serialization; 2. C++ / Python support for broadcasting graph; Python / Dask orchestration for assembling all the functionality above will follow up in a future task, when a few more dependencies will become clearer (`graph_t` object Python wrapper, or opaque handle; decision on whether/when to implement dispatch layer; or even forgo the graph object in python, altogether, and just call a function stub after broadcasting: `distribute_and_call(functor, params)`, etc.); The RW Batch functionality is currently under review. Consequently, the scope of this PR has been modified accordingly. Authors: - Andrei Schaffer (https://github.com/aschaffer) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) - Brad Rees (https://github.com/BradReesWork) URL: #1580
cc @raydouglass @pentschev @quasiben Authors: - https://github.com/jakirkham Approvers: - Peter Andreas Entschev (https://github.com/pentschev) - Ray Douglass (https://github.com/raydouglass) URL: #1634
…icy thrust::seq) from copy_v_transform_reduce_key_aggregated_out_nbr (#1627) thrust::sort(thrust::seq, ....) does not work with arbitrary large input data size and this call can fail if the array size to locally sort is large. This code replaces many thread local sort with one thrust::sort call from host. Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) - Andrei Schaffer (https://github.com/aschaffer) URL: #1627
Similar to rapidsai/cuml#3901. After #1491 and #rapids-cmake and #1585, now at install time, the cugraph headers are being nested into `path/to/env/include/cugraph/cugraph` instead of just `path/to/env/include/cugraph/`. This, as far as I'm aware, is unintentional and unlike the rest of RAPIDS projects (cuDF, RMM and cuML). cc @trxcllnt @robertmaynard Authors: - Dante Gama Dessavre (https://github.com/dantegd) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) - Robert Maynard (https://github.com/robertmaynard) - Seunghwa Kang (https://github.com/seunghwak) - Paul Taylor (https://github.com/trxcllnt) - Rick Ratzel (https://github.com/rlratzel) URL: #1630
Discovered this bug during louvain testing. The variable `idx` is an index controlling iteration over a subset of the vertices, while `major_offset` identifies a particular vertex offset in the overall data structure. Several places where using `idx` where they needed to be using `major_offset`. Authors: - Chuck Hastings (https://github.com/ChuckHastings) Approvers: - Seunghwa Kang (https://github.com/seunghwak) - Andrei Schaffer (https://github.com/aschaffer) URL: #1633
Authors: - https://github.com/Iroy30 Approvers: - Chuck Hastings (https://github.com/ChuckHastings) - Rick Ratzel (https://github.com/rlratzel) URL: #1612
`set_target_properties(cugraphmgtestutil PROPERTIES CUDA_ARCHITECTURES OFF)` was set and this caused cudaFuncGetAttribute to intermittently return an invalid PTX version. This PR deletes this. Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Chuck Hastings (https://github.com/ChuckHastings) - Rick Ratzel (https://github.com/rlratzel) URL: #1638
Fix merge conflicts in 1631
Adds multicolumn support for: jaccard wjaccard overlap woverlap pagerank spectral clustering forceatlas Authors: - https://github.com/Iroy30 Approvers: - Brad Rees (https://github.com/BradReesWork) - Chuck Hastings (https://github.com/ChuckHastings) - Joseph Nke (https://github.com/jnke2016) URL: #1571
[gpuCI] Forward-merge branch-21.06 to branch-21.08 [skip ci]
#1640) Dependent on NVIDIA/cuCollections#83 Authors: - Seunghwa Kang (https://github.com/seunghwak) Approvers: - Brad Rees (https://github.com/BradReesWork) - Chuck Hastings (https://github.com/ChuckHastings) - Rick Ratzel (https://github.com/rlratzel) URL: #1640
[gpuCI] Forward-merge branch-21.06 to branch-21.08 [skip ci]
Upgraded recipe and dev envs to NCCL 2.9.9. The `cugraph` python conda recipe previously included NCCL, but (I think) it's a transient dependency since `libcugraph` actually needs it, so it was removed from the `cugraph` recipe. cc @dantegd for consistency with cuML. Note: the NCCL 2.9.9 upgrade was tested during benchmark runs, but the conda recipe changes have not been tested. Authors: - Rick Ratzel (https://github.com/rlratzel) Approvers: - Jordan Jacobelli (https://github.com/Ethyling) - Chuck Hastings (https://github.com/ChuckHastings) - Seunghwa Kang (https://github.com/seunghwak) URL: #1636
This effort was originally targeted toward the WCC effort, but has been expanded a bit. This supersedes #1545 which I will close. The goal here is to create a means for constructing test graphs in an easier fashion. Testing the capabilities of different graph algorithms might require a variety of graphs. The objective of this PR is to better organize the graph generation components and to introduce some components to help in composing graphs out of multiple components. This PR introduces the following capabilities: * Create an ER graph * Create a collection of Complete Graphs * Create a collection of 2D mesh graphs * Create a collection of 3D mesh graphs * Create a random path graph (connect all vertices with a single randomly ordered path) * Translate vertex ids of a graph * Combine multiple edge lists into a single graph Closes #1543 Authors: - Chuck Hastings (https://github.com/ChuckHastings) Approvers: - Andrei Schaffer (https://github.com/aschaffer) - Brad Rees (https://github.com/BradReesWork) URL: #1603
Closes #1579 Authors: - https://github.com/Iroy30 - Brad Rees (https://github.com/BradReesWork) Approvers: - Brad Rees (https://github.com/BradReesWork) - Andrei Schaffer (https://github.com/aschaffer) - Rick Ratzel (https://github.com/rlratzel) - Joseph Nke (https://github.com/jnke2016) - Chuck Hastings (https://github.com/ChuckHastings) URL: #1599
Should be merged after PR #1636. - Undo NCCL 2.9.6 bug workarounds. - MG WCC weak scaling improvements. Authors: - Seunghwa Kang (https://github.com/seunghwak) - Rick Ratzel (https://github.com/rlratzel) Approvers: - Andrei Schaffer (https://github.com/aschaffer) - Chuck Hastings (https://github.com/ChuckHastings) - Rick Ratzel (https://github.com/rlratzel) URL: #1628
…1648) Made call to cugraph.random_walks() ignore the unused return value. Manually tested in browser with jupyterlab. Author: - Rick Ratzel (https://github.com/rlratzel) Approvers: - Brad Rees (https://github.com/BradReesWork) - AJ Schmidt (https://github.com/ajschmidt8) URL: #1648
Author: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) - Rick Ratzel (https://github.com/rlratzel) - Brad Rees (https://github.com/BradReesWork) URL: #1649
Fix conflicts in `1643`
* fix overflow condition if available memory is larger than 32 bit computation would support * fix clang-format issues
[gpuCI] Forward-merge branch-21.06 to branch-21.08 [skip ci]
[gpuCI] Forward-merge branch-21.06 to branch-21.08 [skip ci]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
aschaffer/21.08 <- rapidsai/21.08