PML UCX: add SPC instrumentation for sent/received message sizes

[devreal]

Sent and received message sizes are currently only tracked for pml/ob1. This PR adds support for tracking message sizes to pml/ucx.

Signed-off-by: Joseph Schuchart schuchart@icl.utk.edu

[bosilca]

This is basically the same level of functionality of PMPI, but without the PMPI. The only interest I can see is that it leaves the PMPI API available for other tools, but in exchange it has to be started manually and cannot be done automatically by the tool.

[devreal]

This is basically the same level of functionality of PMPI, but without the PMPI. The only interest I can see is that it leaves the PMPI API available for other tools

The question for me is: should SPC exposed through MPI_T report metrics as a best effort or should it only report ob1 internals? I would argue that any correct information is better than none and having SPC report information on the application behavior for one PML but not with another PML makes it unreliable and ultimately useless for tools. Yes, I can get the same information through PMPI but one benefit of MPI_T is to provide an easier interface to achieve the same result. As a user, I'd rather query MPI_T than write my own interposition library and I want to see similar statistics with the ucx PML as with the ob1 PML.

in exchange it has to be started manually and cannot be done automatically by the tool.

That is addressed in #8065

[bosilca]

Assuming we go for the lesser capability the approach proposed here will force us to duplicate the code in all PMLs. In same time moving it up to the MPI API would double count the data for all PML that natively support such capability.

[devreal]

There are good reasons to keep the accounting in the PML, probably most importantly to be able to capture messages originating from collectives. Moving the accounting into the MPI layer would also mean that requests have to be tracked to not mess up counting if receives don't complete (because they are cancelled).

It might be desirable to streamline the accounting such that the PMLs provide similar high-level information for sent and received messages, potentially adding PML-specific SPCs reflecting implementation details (the details of how ob1 actually transfers messages using put/get and its control messages may not be interesting to most users and may cause confusion instead). Even then, the arguments for keeping the accounting inside the PMLs and not moving it to the MPI layer still stand. The additional code required in each PML is small and should do no harm.

[bosilca]

The requests that are cancelled have that info in the status, so a correctly implemented OMPI layer can handle this case.