Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update GH man pages #9929

Merged
merged 1 commit into from
Mar 21, 2024
Merged

Update GH man pages #9929

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions main/man/fi_cxi.7.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,18 @@ CXI integrated launcher and CXI authorization key aware libfabric user:
7. Application processes select from the list of available service IDs and VNIs
to form an authorization key to use for Endpoint allocation.

## Endpoint Protocols

The provider supports multiple endpoint protocols. The default protocol is
FI_PROTO_CXI and fully supports the messaging requirements of parallel
applicaitons.

The FI_PROTO_CXI_RNR endpoint protocol is an optional protocol that targets
client/server environments where send-after-send ordering is not required and
messaging is generally to pre-posted buffers; FI_MULTI_RECV is recommended.
It utilizes a receiver-not-ready implementation where
*FI_CXI_RNR_MAX_TIMEOUT_US* can be tuned to control the maximum retry duration.

## Address Vectors

The CXI provider supports both *FI_AV_TABLE* and *FI_AV_MAP* with the same
Expand Down Expand Up @@ -433,6 +445,15 @@ faults but requires all buffers to be backed by physical memory. Copy-on-write
semantics are broken when using pinned memory. See the Fork section for more
information.

The CXI provider supports DMABUF for device memory registration. If the ROCR
and CUDA libraries support it, the CXI provider will default to use DMA-buf.
There may be situations with CUDA that may double the BAR consumption.
Until this is fixed in the CUDA stack, the environment variable
*FI_CXI_DISABLE_DMABUF_CUDA* can be used to fall back to the nvidia
peer-memory interface.
Also, *FI_CXI_DISABLE_DMABUF_ROCR* can be used to fall back to the amdgpu
peer-memory interface.

## Translation Cache

Mapping a buffer for use by the NIC is an expensive operation. To avoid this
Expand Down Expand Up @@ -1077,6 +1098,12 @@ The CXI provider checks for the following environment variables:
*FI_CXI_DEFAULT_VNI*
: Default VNI value used only for service IDs where the VNI is not restricted.

*FI_CXI_RNR_MAX_TIMEOUT_US*
: When using the endpoint FI_PROTO_CXI_RNR protocol, this setting is used to
control the maximum time from the original posting of the message that the
message should be retried. A value of 0 will return an error completion
on the first RNR ack status.

*FI_CXI_EQ_ACK_BATCH_SIZE*
: Number of EQ events to process before writing an acknowledgement to HW.
Batching ACKs amortizes the cost of event acknowledgement over multiple
Expand Down
Loading