Adds sparse communicator class

[MarcelKoch]

Based on #1544

This PR adds a class that handles the communication of a local vector. It uses the neigborhood non-blocking all-to-all for this. The sparse comm is constructed from an index_map.

In another PR this will be used in the distributed matrix.

Note: This uses C++17 to use type-erasure for the sparse_communicator.

PR Stack:

• Add segmented array type #1545
• Distributed Index Map #1543
• Adds Index map device kernels #1579
• Use index_map in distributed::matrix #1544
• Allow to optionally use C++17 #1548
• Adds sparse communicator class #1546
• Use sparse communicator in distributed::Matrix #1549

[MarcelKoch]

@upsj I think this PR might be a good opportunity to discuss your distributed::RowGather idea. In principle this class is already that, but not in the lin-op hierarchy.

[pratikvn]

I think this class structure mixes a few things together: the communicator class idea, with the specific dense communicate (involving a all_to_all_v). I think it would make sense to separate the two.

I think it should be possible to make the sparse_communicator a first class communicator and overload only those functions as necessary, for example, all_to_all_v in this case. The distributed object can then just use sparse communicator as a normal communicator and inherits the better all_to_all_v as it uses the sparse communicator for communication. For all other functions where the sparse communicator does not make sense (MPI_Send etc), you transparently use the underlying default communicator.

[pratikvn]

If it only provides a neighborhood all to all communication, then I think calling it a sparse communicator might be misleading.

[pratikvn]

A public method of a public class should not be taking objects within the detail namespace. Probably need to generalize it to array or Dense ?

[MarcelKoch]

The issue is that if the communicator stores the caches (which I also prefer) then the communicator can handle only one async communication at a time. Any new communication would need to wait until the internal buffers are ready to access.

[pratikvn]

Why not

Suggested change

	MPI_Info info;
	GKO_ASSERT_NO_MPI_ERRORS(MPI_Info_create(&info));
	GKO_ASSERT_NO_MPI_ERRORS(MPI_Dist_graph_create_adjacent(
	comm.get(), send_ids.get_size(), send_ids.get_const_data(),
	MPI_UNWEIGHTED, recv_ids.get_size(), recv_ids.get_const_data(),
	MPI_UNWEIGHTED, info, false, &sparse_comm));
	GKO_ASSERT_NO_MPI_ERRORS(MPI_Info_free(&info));
	GKO_ASSERT_NO_MPI_ERRORS(MPI_Dist_graph_create_adjacent(
	comm.get(), send_ids.get_size(), send_ids.get_const_data(),
	MPI_UNWEIGHTED, recv_ids.get_size(), recv_ids.get_const_data(),
	MPI_UNWEIGHTED, MPI_INFO_NULL, false, &sparse_comm));

since no key-value pairs are being set anyway ?

[MarcelKoch]

One MPI implementation, I can't remember on which system, crashed without it, so I just kept it.

[pratikvn]

Looking at this now, I dont get why the sparse_communicator class handles this specific communicate function.

[MarcelKoch]

@pratikvn you are right, the class mixes a few responsibilities. I will split it up into one class that handles the all-to-all communication and another one which does the distributed row gather.

[MarcelKoch]

This is superseded by #1588 and #1589