ch4/ofi: ssend reply is not safe with multiple messages

[hzhou]

In the native path, ssend receiver sends ack to the sender via a direct tag match. When the sender sends multiple ssends to the receiver with the same tag, the reply is not guaranteed to match the original send.

The current multi-thread ssend tests in test/mpi/threads/pt2pt is insufficient to catch this issue because the test doesn't really check the semantics of send -- difficult to check. The test passes as long as all the ssend gets ack'ed regardless of mismatching.

TODO

• make a test confirm this issue
  * maybe we can send multiple messages both large and small all with the same tag, and measure the Wtime of the ssend. If sending small message takes longer than large message, then we know it is wrong.
  ❌ different threads contend and the timing isn't reliable.

[raffenet]

I guess it depends on completion order on the recv side? The way we ensure acks go to the right place in AM code is to use the send request handle. Could we do that here by setting the match bits to the request handle value? Or is there an issue with fitting all bits in the available tag space?

[hzhou]

There are 64 bits in the matchbits, lower 32 bits for tag, up to 20 bits for context id, then at least 4 bits for misc flags, so we only have 8 bits left, which isn't enough to hold request handle.

I am thinking that when cq_data size is 8 -- seems typical, we could use cq_data for both the source rank and request handle. I am experimenting this in #5575

[raffenet]

You don't need context id or tag in this case. Just sync bit and request handle, right?

[hzhou]

You don't need context id or tag in this case. Just sync bit and request handle, right?

Oh, it's about sending the sreq handle to the receiver in the first place. If receiver doesn't have the sreq handle, it can't use it in the ack.

[raffenet]

You don't need context id or tag in this case. Just sync bit and request handle, right?

Oh, it's about sending the sreq handle to the receiver in the first place. If receiver doesn't have the sreq handle, it can't use it in the ack.

Oh, right. I get it now.

[wesbland]

There are also some cases where you don't get all 64 bits. I don't remember bits bits we use for a request handle.

[hzhou]

There are also some cases where you don't get all 64 bits. I don't remember bits bits we use for a request handle.

Yes. For those cases, we have to resort to direct tag matching. But if we have the extra cq data bits, we can use it for more robustness. In PR #5575, we choose the mechanism based on runtime cq_data_size.

[hzhou]

When multiple ssend with the same tags is on the fly, because all the SYNC packets coming back are identical, so it doesn't really matter which one is going to which, as long as in the end all the SYNC packets will be accounted for. The timing will be off, but in the end all will get completed. I don't think there is anything the user can detect.