How to ensure atomic write operation by specifying 8 bytes of data?

[yangx-jy]

Hi,

rpma_write_atomic() just passes 8 bytes of data to rpma_mr_write(), so does the atomic write operation depend on RDMA write with 8 bytes? If wrong, How to ensure atomic write operation by specifying 8 bytes of data?

Best Regards,
Xiao Yang

[grom72]

Hi Xiao Yang,
Atomicity in RDMA is supported explicitly by atomic verbs Cmp/Swap and Fetch/Add, which are very heavy operations.
There is an ongoing standardization effort to add Atomic Write as a new, separate verb to RDMA API.
Without native support in HW, an atomic write could be implemented as a Read and a Cmp/Swap operations sequence. This is a very long and heavy procedure, but it should guaranty 100% atomicity.
Nowadays, all known RDMA NICs support 8 bytes atomicity for aligned memory. The same is for PCIe write operation (at least on Intel platform) - this is the foundation of the present implementation of atomic write in the librpma. Recently we also improved implementation by adding a fencing mechanism (#603) that makes the librpma atomic write almost compatible with what IBTA Spec defines.
Regards
Tomasz

[ykorman]

@grom72, the article referred to in your readme, seems to contradict your statement:

IA server ensures 8 bytes atomicity when accessing persistent memory locally. That is not valid for the RDMA Write operation. The only way to ensure 8 bytes atomicity in case of remote Write to persistent memory is to do this via a dedicated software component in the Target node.

What changed since the article?

Thanks,
Yigal

[grom72]

Hi Yigal,
officially nothing has changed since the article. None of the RDMA NIC vendors could confirm 8 bytes write atomicity officially, as IBTA Spec does not ensure write operation atomicity. But when considering implementation details, it is impossible to cut 8-byte aligned write operation into more than one, neither on RDMA nor PCIe level. And this is the present solution supported by the librpma. It will change in the future with new verbs defined by IBTA and if there will be such a need, we also consider building an alternative "legacy" approach based on the read cmp/swap verbs.
The whitepaper you have mentioned has been based on librpmem solution where 8 bytes atomicity has been implemented with send/recv. But in the librpma, we have decided to leave send/recv verbs for an application and not use it at all on the library level. So application could either use librpma API for atomic write that should work fine on the majority of HW solution or build it's own on top of it using send/recv services.

[ykorman]

I understand, thank you for explaining.