core: handle dmabuf for ofi_mr functions #9499
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Intel CI error is probably transient, will retry. |
|
What about |
mr cache is currently skipped for dmabuf reg |
|
@j-xiong do you think we should a similar convert from dmabuf to iov for |
|
@shijin-aws Yes, I think it makes sense to support this in mr cache for the case that virtual address mapping is available (i.e. base_addr != 0). |
Will update, thanks |
shijin-aws
force-pushed
the
ofi_mr_dmabuf
branch
from
131a166 to
af4d260
Compare
|
@j-xiong please see the updated commits. I didn't want to check whether |
j-xiong
reviewed
Nov 3, 2023
include/ofi_mr.h
Outdated
| @@ -427,9 +438,9 @@ int ofi_mr_cache_search(struct ofi_mr_cache *cache, | |||
| * with the cache. | |||
| */ | |||
| struct ofi_mr_entry *ofi_mr_cache_find(struct ofi_mr_cache *cache, | |||
| const struct fi_mr_attr *attr); | |||
| const struct fi_mr_attr *attr, uint64_t flags); | |||
There was a problem hiding this comment.
Line is too long. Break after the comma.
prov/util/src/util_mr_map.c
Outdated
| { | ||
| cur_abi_attr->mr_iov = (struct iovec *) user_attr->mr_iov; | ||
| if (FI_VERSION_GE(user_version, FI_VERSION(1, 20)) && (flags & FI_MR_DMABUF)) |
include/ofi_mr.h
Outdated
| int i; | ||
|
|
||
| for (i = 0; i < count; i++) { | ||
| iov[i].iov_base = (void *) ((uintptr_t) dmabuf[i].base_addr + dmabuf[i].offset); |
prov/efa/src/efa_mr.c
Outdated
| @@ -634,7 +634,7 @@ int efa_mr_is_cuda_memory_freed(struct efa_mr *efa_mr, bool *freed) | |||
| * negative libfabric error code on failure. | |||
| */ | |||
| static | |||
| int efa_mr_update_domain_mr_map(struct efa_mr *efa_mr, struct fi_mr_attr *mr_attr) | |||
| int efa_mr_update_domain_mr_map(struct efa_mr *efa_mr, struct fi_mr_attr *mr_attr, uint64_t flags) | |||
prov/efa/src/efa_mr.c
Outdated
| @@ -750,14 +750,14 @@ int efa_mr_update_domain_mr_map(struct efa_mr *efa_mr, struct fi_mr_attr *mr_att | |||
| } | |||
| #else /* HAVE_CUDA */ | |||
| static | |||
| int efa_mr_update_domain_mr_map(struct efa_mr *efa_mr, struct fi_mr_attr *mr_attr) | |||
| int efa_mr_update_domain_mr_map(struct efa_mr *efa_mr, struct fi_mr_attr *mr_attr, uint64_t flags) | |||
shijin-aws
force-pushed
the
ofi_mr_dmabuf
branch
from
af4d260 to
5c1e891
Compare
When a mr is registered with FI_MR_DMABUF, mr_iov is replaced by the fi_mr_dmabuf object in the mr_attr. When inserting the mr to mr map, libfabric needs to translate the dmabuf object to mr_iov. Signed-off-by: Shi Jin <sjina@amazon.com>
When mr is registered with FI_MR_DMABUF bit in the flag, ofi_mr_attr_update should pass the fi_mr_dmabuf object from the attr user passed Signed-off-by: Shi Jin <sjina@amazon.com>
Convert dmabuf to iov for ofi_mr_info in ofi_mr_cache_find and ofi_mr_cache_reg. When constructing ofi_mr_info from mr_attr for ofi_mr_cache_search. Convert dmabuf to iov if the flags has FI_MR_DMABUF. Signed-off-by: Shi Jin <sjina@amazon.com>
shijin-aws
force-pushed
the
ofi_mr_dmabuf
branch
from
5c1e891 to
4aa6577
Compare
|
bot:aws:retest |
|
@j-xiong ok to merge? |
j-xiong
approved these changes
Nov 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When mr is registered with FI_MR_DMABUF bit in the flag,
ofi_mr_attr_update should pass the fi_mr_dmabuf object
from the attr user passed.
When inserting the mr to mr map, libfabric
needs to translate the dmabuf object to mr_iov.
When constructing ofi_mr_info from mr_attr for ofi_mr_cache_search.
Convert dmabuf to iov if the flags has FI_MR_DMABUF.