Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH : adding parallel implementations of all_pairs_ algos #33

Merged
merged 24 commits into from
Mar 26, 2024

Conversation

Schefflera-Arboricola
Copy link
Member

@Schefflera-Arboricola Schefflera-Arboricola commented Jan 17, 2024

Please do "Squash and merge" while merging this PR

Added the following :

  • all_pairs_dijkstra
  • all_pairs_dijkstra_path_length
  • all_pairs_dijkstra_path
  • all_pairs_bellman_ford_path_length
  • all_pairs_shortest_path
  • all_pairs_shortest_path_length - no speedups(the standard sequential implementation is already really fast, and even when the graph size was increased till 4000 nodes, there were negligible speedups, and it only took a minute or 2 to generate its heatmap)
  • all_pairs_node_connectivity1
  • approximate_all_pairs_node_connectivity1
  • all_pairs_all_shortest_paths


[1] here, default chunking means having chunks of at most 10 nodes, because otherwise, it was taking too long with bigger node_chunks; and I also reduced the number of nodes while timing, but the speedups are pretty consistent(i.e. around 3). This heatmap alone took 2.5 hrs, and ran successfully after several tries. Also no heatmap for approximate_all_pairs_node_connectivity because for a random graph, it will result in the same heatmap as all_pairs_node_connectivity

resolved conflicts(rebased)
@Schefflera-Arboricola Schefflera-Arboricola marked this pull request as draft January 17, 2024 12:22
@Schefflera-Arboricola Schefflera-Arboricola changed the title adding parallel implementations in weighted.py adding parallel implementations of all_pairs_ algos Jan 23, 2024
@Schefflera-Arboricola Schefflera-Arboricola marked this pull request as ready for review January 23, 2024 17:12
@Schefflera-Arboricola Schefflera-Arboricola changed the title adding parallel implementations of all_pairs_ algos ENH : adding parallel implementations of all_pairs_ algos Jan 23, 2024
resolved conflicts(rebased)
@dschult dschult added the type: Enhancement New feature or request label Jan 30, 2024
@Schefflera-Arboricola Schefflera-Arboricola marked this pull request as draft February 21, 2024 18:25
…tivity and added connectivity.connectivity.all_pairs_node_connectivity and all_pairs_all_shortest_paths
…d an if-else statement, added and updated heatmaps(no speedups for all_pairs_shortest_path_length)
@Schefflera-Arboricola Schefflera-Arboricola marked this pull request as ready for review February 26, 2024 20:37
dschult and others added 2 commits March 10, 2024 20:35
Co-authored-by: Dan Schult <dschult@colgate.edu>
@Schefflera-Arboricola
Copy link
Member Author

Tests / test (macos-latest, 3.11) is running for more than 4 hrs. It is stuck at:

algorithms/centrality/tests/test_group.py ........................       [ 50%]

@dschult
Copy link
Member

dschult commented Mar 11, 2024

That happened to me yesterday too. I don’ think it has anything to do with the backend, but I’m not sure.

@dschult
Copy link
Member

dschult commented Mar 15, 2024

Where does this PR stand? Is there more to do here?
Do you feel the improvements here are what you expect -- or are there other questions/issues you want to explore with this PR?

@Schefflera-Arboricola
Copy link
Member Author

I think we can merge this. The only algo not showing any speedups is all_pairs_shortest_path_length and its sequential implementation is already really fast. But, we should think about how to handle such functions in the whole of nx-parallel, i.e the functions that don’t show any speedups for the general random graphs we use in the current timing script and/or whose sequential networkx implementation is already really fast(like number_of_isolates, all_pairs_shortest_path_length, global_reaching_centrality and local_reaching_centrality(#44), etc.). For some such algos setting the minimum chunk size to be really large might help(this didn't work for all_pairs_shortest_path_length), but then the input graph should also be really big. Just a thought, but having something like should_run = True only when the graph is really big might be helpful.

@dschult dschult merged commit 84d5635 into networkx:main Mar 26, 2024
11 checks passed
@jarrodmillman jarrodmillman added this to the 0.1 milestone Mar 26, 2024
@jarrodmillman jarrodmillman modified the milestones: 0.1, 0.2 May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: Enhancement New feature or request
Development

Successfully merging this pull request may close these issues.

None yet

3 participants