Use index_map in distributed::matrix #1544

MarcelKoch · 2024-02-14T16:50:31Z

Application of #1543. Now the index_map is used to create the numbering for the non-local columns. The index_map is created during the distributed::matrix::read_distributed and also returned from that function. This can allow users to provide already separated local and non-local matrices for the read_distributed further down the line.
Also, it makes the mapping non-local -> global more easily available.

This PR also introduces:

making run easier to use
specialization of run for distributed matrices -> removes need for changes to distributed base

PR Stack:

MarcelKoch · 2024-02-14T16:51:48Z

There is some interface breaking stuff in there, but it's minor and I can easily add the old version, if necessary.

pratikvn

LGTM! The issue of interface needs to be resolved. It is in the experimental namespace, but I think we said we would still not break interface in experimental ?

pratikvn · 2024-04-01T09:46:12Z

include/ginkgo/core/distributed/matrix.hpp

     */
-    void read_distributed(
+    index_map<local_index_type, global_index_type> read_distributed(


this breaks interface

we made no promises about interface-stability in experimental beyond "we'll try to keep changes to a minimum"

TBH, I don't consider that really an interface break. This has an effect only on very serious template meta programming, which I don't expect users to do. Other than that, current code will compile, it just discards the return value.

There is actually an interface break here, but not due to the return value. Now this function expects a shared_ptr<Partition> instead of a raw pointer. But this is done in #1543 if you want to discuss that.

Somehow the old comments I made dont appear anymore, so it is a bit annoying

But this is indeed an unfortunate interface break. I think the version with shared_ptr should be okay, but I think users actually depend on read_distributed, so it will break existing code.

I think the index map needs to store the partition. I've looked again into the map_to_local kernels, and there I actually need all arrays a partition stores. So it wouldn't make sense to only copy the needed data out of the partition and into the index-map.

sonarcloud · 2024-05-10T21:01:31Z

Quality Gate passed

Issues
19 New issues
0 Accepted issues

Measures
0 Security Hotspots
92.3% Coverage on New Code
3.6% Duplication on New Code

See analysis details on SonarCloud

upsj

LGTM, minor gripe with the memory allocation tracking that is incomplete in index_map, maybe a fix could be included here?

upsj · 2024-05-17T07:06:46Z

benchmark/test/reference/spmv_distributed.profile.stdout

@@ -5,7 +5,7 @@
        "comm_pattern": "stencil",
        "spmv": {
            "csr-csr": {
-                "storage": 6564,
+                "storage": 6420,


Maybe not directly related to this, but I went back to check, there are some places left over in #1543 that still use std::vector instead of gko::vector or gko::array.

I've only checked (reference|omp)/distributed for std::vector and found only 4 instances. Did you find more? (I'm updating them btw)

no, I checked the patch, that should be all. The rest happens inside tests

I switched to gko::vector and it didn't change the benchmark output (because the missing changes were only in omp/).

I know, this only reports the resting storage size, not temporary memory. But it did remind me to check for missing updates :)

I see. Maybe also providing the maximal storage might be interesting to add in the future.

Yes, good point!

upsj · 2024-05-17T07:16:32Z

core/distributed/matrix.cpp

+    for (size_type i = 0; i < host_recv_targets->get_size(); ++i) {
+        recv_sizes_[host_recv_targets->get_const_data()[i]] =
+            host_offsets->get_const_data()[i + 1] -
+            host_offsets->get_const_data()[i];
+    }


It feels pedantic, but I think having a raw loop next to a bunch of algorithm calls is a mismatch of abstraction levels - this might be better extracted into a separate function, similar to Sean Parent's "no raw loops" suggestion (https://www.youtube.com/watch?v=W2tWOdzgXHA)

Maybe we can call it compute_recv_sizes? Then the high-level control flow is easy to follow, and if you need to know how those recv_sizes get determined, you look at the function call.

IMO the issue here is that arrays can't be used with STL. If we had begin, and end for them, this would just be std::adjacent_difference

You can use pointers here , but adjacent_difference is not 100% sufficient, since it leaves the first element unchanged and doesn't scatter the results. I just meant extract these 5 lines into a function

You're right, adjacent_difference is weird, same as partial_sum. I would still argue that the issue here is just that our arrays are not easy to access. If it would read

for (size_type i = 0; i < ecv_targets.size(); ++i) { recv_sizes_[recv_targets[i]] = offsets[i + 1] - offsets[i]; }

I don't think you would have the same issue. Only because we have to clutter it with host_ prefixes, and use very verbose function names it becomes an issue.

My comment is mostly not about the complexity of the code (.get_const_data()[] vs []) but about mixing low-level code with high-level code. I do agree that it would be nice to have a cleaner access to arrays. If your extracted function takes pointers, you would arrive at that code though.

Maybe we could add a host_array type that is derived from an array and a custom constructor that always takes get_master(), with additional operator[] and begin/end() functions?

I've changed this, but maybe not exactly how you imagined it. I'm using a lambda for this, since I don't want to put it outside the function, due to it's limited use.

upsj · 2024-05-17T07:23:41Z

reference/test/distributed/matrix_kernels.cpp

-        {std::make_tuple(gko::dim<2>{2, 1}, I<git>{1}, I<git>{0}, I<vt>{6}),
-         std::make_tuple(gko::dim<2>{2, 3}, I<git>{0, 0, 1}, I<git>{2, 1, 0},
+        {std::make_tuple(gko::dim<2>{2, 1}, I<git>{1}, I<git>{1}, I<vt>{6}),
+         std::make_tuple(gko::dim<2>{2, 3}, I<git>{0, 0, 1}, I<git>{1, 3, 2},


These changes are a bit hard to review - can you add a comment to the function explaining what the tuple members represent?

MarcelKoch · 2024-05-17T07:34:25Z

core/distributed/matrix.cpp

+    auto imap = index_map<local_index_type, global_index_type>(
+        exec, col_partition, comm.rank(), global_non_local_col_idxs);


I'm quite confident that it will become necessary to store this map in the matrix class in the future. I'm not doing that for now, because with the distributed PGM, there are now ways to create a non-empty matrix without this read function. I don't know how to create the mapping in the PGM case, so there would be non-empty matrices, but with empty mappings.
I didn't think this is suitable, so I don't store the mapping. But making it a member in the future would either require unnecessary copies or an interface break.
Any thoughts on this @upsj @pratikvn?

I think it's a fundamental choice - do we want to consider the matrix storing the implicitly block-reordered matrix (every rank has a consecutive range of indices), or the original matrix? I'm fine with the latter, but it may complicate things as mappings global <--> local are more costly. But at the same time, I'm not sure that would ever be a performance bottleneck.

Can you elaborate on the interface break?

interface break:

auto imap = mat->read_distributed(...);

will result in an error if read_distributed returns again void. The return type could be changed to const index_map& but this would result in copies.

Without the original matrix, we can't do anything that involves more than the local block. So, the distributed PGM would not work, also RAS would not be possible. Also I think the mapping is necessary to provide users an option to use pre-existing data, since it allows mapping global indices, which the users know, to the ginkgo non-local indices, which the user doesn't know.

That's not entirely true - you can operate on the block-reordered indexing just fine, imagine computing an index_map after remapping each rank's indices to be consecutive. But that is still doing 90% of the work towards having a full global <--> local index map, so it's probably to better to do it fully.

IMO, just from a properties perspective, I think it makes sense for the matrix class to have an index map. Any function that creates a distributed matrix, should then also be required to update its index_map. I understand that we dont do that for PGM, right now, but I think the interface choices should not be dependent on that.

So, I think it should be okay to store an index_map in the matrix class.

TBH, I am not sure why read_distributed should return an index_map. I think we should just store it inside the class and have a public getter to access it where necessary.

I think the reordering question is a bit separate from the index-map question. As you said, you could still have an index map after reordering. In general, I think reordering can become quite complicated, see the fun image in petsc's user guide: https://petsc.org/release/manual/mat/#block-matrices

I think for now I will just remove the return type. The index map is still helpful, since it simplifies the matrix kernels, while the information necessary for PGM is still available. Thus we could add the index map as a member at a later point if we want to.

- comment tests - replace std::vectors in backend - refactoring Co-authored-by: Tobias Ribizel <mail@ribizel.de>

MarcelKoch self-assigned this Feb 14, 2024

ginkgo-bot added reg:testing This is related to testing. mod:all This touches all Ginkgo modules. labels Feb 14, 2024

MarcelKoch mentioned this pull request Feb 14, 2024

Adds sparse communicator class #1546

Closed

7 tasks

MarcelKoch force-pushed the index-map branch from c1ecb3c to c49b4b7 Compare February 15, 2024 12:51

MarcelKoch force-pushed the read-distributed-with-index-map branch from 58c6462 to 11f9ba3 Compare February 15, 2024 12:51

MarcelKoch mentioned this pull request Feb 15, 2024

Allow to optionally use C++17 #1548

Closed

7 tasks

MarcelKoch force-pushed the read-distributed-with-index-map branch from 11f9ba3 to 12f8ab3 Compare March 25, 2024 10:27

MarcelKoch force-pushed the index-map branch from c49b4b7 to 6a173a4 Compare March 25, 2024 10:27

MarcelKoch force-pushed the read-distributed-with-index-map branch from 12f8ab3 to c12bac1 Compare March 25, 2024 10:31

MarcelKoch force-pushed the index-map branch from 6a173a4 to a7d1dab Compare March 25, 2024 10:31

This was referenced Mar 25, 2024

Distributed Index Map #1543

Merged

Adds Index map device kernels #1579

Merged

Add segmented array type #1545

Merged

MarcelKoch changed the base branch from index-map to index-map-device-kernels March 25, 2024 10:36

MarcelKoch mentioned this pull request Mar 25, 2024

Use sparse communicator in distributed::Matrix #1549

Closed

7 tasks

pratikvn self-requested a review March 25, 2024 10:54

pratikvn requested changes Apr 1, 2024

View reviewed changes

MarcelKoch force-pushed the read-distributed-with-index-map branch from c12bac1 to a1d45ac Compare April 2, 2024 07:37

MarcelKoch force-pushed the index-map-device-kernels branch from 8d97aef to 2abe709 Compare April 2, 2024 07:37

MarcelKoch force-pushed the read-distributed-with-index-map branch from a1d45ac to 07f6d02 Compare April 2, 2024 17:31

MarcelKoch force-pushed the index-map-device-kernels branch 2 times, most recently from 8206bc3 to c700b1c Compare April 4, 2024 10:29

MarcelKoch force-pushed the read-distributed-with-index-map branch from 07f6d02 to b42ab92 Compare April 4, 2024 10:29

MarcelKoch force-pushed the index-map-device-kernels branch from c700b1c to 989967d Compare April 4, 2024 11:00

MarcelKoch force-pushed the read-distributed-with-index-map branch from b42ab92 to 8f104fd Compare April 4, 2024 11:00

MarcelKoch added this to the Ginkgo 1.8.0 milestone Apr 5, 2024

MarcelKoch force-pushed the read-distributed-with-index-map branch from 8f104fd to a0824a8 Compare April 19, 2024 14:39

MarcelKoch force-pushed the index-map-device-kernels branch from 989967d to cc997ff Compare April 19, 2024 14:39

MarcelKoch force-pushed the index-map-device-kernels branch from 80e2802 to 33d28fb Compare May 9, 2024 07:55

MarcelKoch force-pushed the read-distributed-with-index-map branch from 2072219 to 20e3d96 Compare May 9, 2024 07:55

MarcelKoch force-pushed the index-map-device-kernels branch from 33d28fb to cdb0447 Compare May 9, 2024 08:56

MarcelKoch force-pushed the read-distributed-with-index-map branch from 20e3d96 to 2812123 Compare May 9, 2024 08:56

MarcelKoch requested a review from upsj May 10, 2024 09:24

MarcelKoch force-pushed the index-map-device-kernels branch from cdb0447 to abadebc Compare May 13, 2024 09:41

MarcelKoch force-pushed the read-distributed-with-index-map branch from 2812123 to 52dcfc8 Compare May 13, 2024 09:41

MarcelKoch force-pushed the index-map-device-kernels branch from abadebc to 8fb9d44 Compare May 13, 2024 09:43

MarcelKoch force-pushed the read-distributed-with-index-map branch 2 times, most recently from 1c0bd5f to 355e776 Compare May 13, 2024 20:50

MarcelKoch force-pushed the index-map-device-kernels branch from 8fb9d44 to fdb3ce2 Compare May 13, 2024 20:50

MarcelKoch force-pushed the read-distributed-with-index-map branch from 355e776 to 204d7e8 Compare May 14, 2024 08:20

MarcelKoch force-pushed the index-map-device-kernels branch from fdb3ce2 to dff0aa8 Compare May 14, 2024 08:20

Base automatically changed from index-map-device-kernels to develop May 14, 2024 13:13

MarcelKoch force-pushed the read-distributed-with-index-map branch from 204d7e8 to 3be79ab Compare May 14, 2024 13:25

upsj approved these changes May 17, 2024

View reviewed changes

MarcelKoch commented May 17, 2024

View reviewed changes

MarcelKoch added 2 commits May 17, 2024 08:31

[dist-mat] introduce kernel to separate local and non-local matrix data

da99190

[dist-mat] use index_map in matrix::read_distributed

2eb2ff9

MarcelKoch force-pushed the read-distributed-with-index-map branch from 3be79ab to b519e56 Compare May 17, 2024 13:42

MarcelKoch and others added 4 commits May 17, 2024 15:07

[dist-mat] adjust usages of read_distributed

7d75f3d

[dist-mat] don't return an index map from read-distributed

be28352

[dist-mat] review updates:

277a8fb

- comment tests - replace std::vectors in backend - refactoring Co-authored-by: Tobias Ribizel <mail@ribizel.de>

[bench] update benchmark tests with new output

0d3b710

MarcelKoch force-pushed the read-distributed-with-index-map branch from b519e56 to 0d3b710 Compare May 17, 2024 15:07

MarcelKoch added 1:ST:ready-to-merge This PR is ready to merge. and removed 1:ST:ready-for-review This PR is ready for review labels May 17, 2024

MarcelKoch merged commit a61989f into develop May 20, 2024
11 of 15 checks passed

MarcelKoch deleted the read-distributed-with-index-map branch May 20, 2024 13:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use index_map in distributed::matrix #1544

Use index_map in distributed::matrix #1544

MarcelKoch commented Feb 14, 2024 •

edited

Loading

MarcelKoch commented Feb 14, 2024

pratikvn left a comment

pratikvn Apr 1, 2024

upsj Apr 4, 2024 •

edited

Loading

MarcelKoch Apr 4, 2024

MarcelKoch Apr 30, 2024

pratikvn May 7, 2024

pratikvn May 7, 2024

MarcelKoch May 7, 2024

sonarcloud bot commented May 10, 2024

upsj left a comment

upsj May 17, 2024

MarcelKoch May 17, 2024

upsj May 17, 2024

MarcelKoch May 17, 2024

upsj May 17, 2024

MarcelKoch May 17, 2024

upsj May 17, 2024

upsj May 17, 2024

MarcelKoch May 17, 2024

upsj May 17, 2024

MarcelKoch May 17, 2024

upsj May 17, 2024 •

edited

Loading

MarcelKoch May 17, 2024

upsj May 17, 2024

MarcelKoch May 17, 2024

upsj May 17, 2024

upsj May 17, 2024

MarcelKoch May 17, 2024

MarcelKoch May 17, 2024

upsj May 17, 2024

pratikvn May 17, 2024

MarcelKoch May 17, 2024

MarcelKoch May 17, 2024

		auto imap = index_map<local_index_type, global_index_type>(
		exec, col_partition, comm.rank(), global_non_local_col_idxs);

Use index_map in distributed::matrix #1544

Use index_map in distributed::matrix #1544

Conversation

MarcelKoch commented Feb 14, 2024 • edited Loading

MarcelKoch commented Feb 14, 2024

pratikvn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

upsj Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented May 10, 2024

Quality Gate passed

upsj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

upsj May 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcelKoch commented Feb 14, 2024 •

edited

Loading

upsj Apr 4, 2024 •

edited

Loading

upsj May 17, 2024 •

edited

Loading