Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

osc/rdma: Handle remote completion behavior #9975

Merged
merged 13 commits into from
Feb 16, 2022
Merged

osc/rdma: Handle remote completion behavior #9975

merged 13 commits into from
Feb 16, 2022

Commits on Feb 16, 2022

  1. btl/am-rdma: Pass the correct pointer to get 

    The put over get implementation passed the wrong pointer to the
cbdata of the get call, resulting in all kinds of badness when
the get completed.  This patch passes through the expected pointer.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    8fedcc0 View commit details
    Browse the repository at this point in the history

  2. osc/rdma: Clean up initialization debugging 

    Clean up language around btl selection phases to match rest of the
code (accelerated/alternate).

Also switch component initialization code to use opal_output_verbose
rather than OSC_RDMA_VERBOSE, so that the debugging prints can
be enabled on a non-debug build.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    b6b16a6 View commit details
    Browse the repository at this point in the history

  3. osc/rdma: Load all btls as alternate btls 

    See lengthy comment in the change, but this patch removes the
ability of users to specify a subset of available btls for use
by the osc rdma component.  The BTL interface was never designed
for such usage (which is why there is no similar option for the
OB1 PML) and it had clear places where it broke, so remove it.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    61aca9e View commit details
    Browse the repository at this point in the history

  4. osc/rdma: Simplify query logic 

    With the change to load all btls in the alternate case and the
removal of the logic to change priority based on alternate or
accelerated btls, there's no point in running the full btl
query in osc_rdma_component_query().  Instead, just verify
that there is some number of BTLs available, which is the
extent of testing in the alternate case.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    e93f041 View commit details
    Browse the repository at this point in the history

  5. osc/rdma: Refactor btl wrapper code 

    Refactor wrapper calls around the btl atomic code into their own
header, removing them from the lock interface header.  Clean up
some resulting header dependency changes.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    0394543 View commit details
    Browse the repository at this point in the history

  6. osc/rdma: Wrap calls to BTL RDMA functiions 

    Add wrappers to call all BTL RDMA functions.  This commit just adds
another layer of indirection to the communication calls.  However,
a follow-on patch will add logic to either call the BTL RDMA
functions directly (for accelerated mode) or call the BTL AM RDMA
compatibility layer for alternate mode.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    6a15883 View commit details
    Browse the repository at this point in the history

  7. osc/rdma: Split btl storage based on mode 

    Split the storage of btls based on whether the window is using
accelerated or alternate btls.  This makes it more obvious when
the code has made assumptions about mode that may not be true
(such as the memory registration calls throughout the code that
assumed selected_btl[0] was the one true BTL).

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    75e36e5 View commit details
    Browse the repository at this point in the history

  8. osc/rdma: Remove duplicate op emulation check 

    ompi_osc_rdma_btl_op() already includes a check to emulate a remote
op with a remote fetch-and-op, so there's no need for a second check
in ompi_osc_rdma_acc_single_atomic().

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    222b7f3 View commit details
    Browse the repository at this point in the history

  9. osc/rdma: Remove usage of custom min() function 

    Use the opal_min() function instead of a custom macro.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    6e2af18 View commit details
    Browse the repository at this point in the history

  10. osc/rdma: Clean up use_cpu_atomics behaivor 

    The use_cpu_atomics member of the rdma module was only used during
initialization, so there was no reason for it to be a module
member.

Also clean up the initialization logic for the field (and therefore,
whether or not the shared state optimization is used).  Previously,
it appeared to be possible to use with alternate btls across multiple
nodes (ie, we tested the GLOB value on the btls), but the
use_cpu_atomics field was initialized to false in the multi-node case
and so they would never actually be enabled.  Make that more obvious,
rather than try to enable the optimization, to reduce the risk of
introducing new datapath bugs.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    94fed3b View commit details
    Browse the repository at this point in the history

  11. osc/rdma: Use BTL am-rdma explicit interface 

    Switch from using the implicit BTL interface (where the am-rdma
interface just extends missing functionality in the BTL) to the
new explicit interface (where the OSC RDMA interface is the
only maintainer of the BTL list.

With this change, alternate BTLs do not have to support
REMOTE_COMPLETION to be selected (because the AM RDMA interface
always provides remote completion when we request it, as this
patch does).  Any BTL that supports Active Messages (ie, all of
them) should be able to support the OSC RDMA required semantics,
eliminating the problem of creating windows with no servicable
BTLs.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    4080d5d View commit details
    Browse the repository at this point in the history

  12. osc/rdma: Lower priority to 20 

    The RDMA OSC component is the "general" component, similar to OB1.
The OSC subsystem has gone through some priority inflation over
the years, so try to clean that up. This patch lowers the RDMA
OSC component priority from 101 to 20, leaving the priorities as:

  sm      100
  ucx     60 (if API version > 1.5.0)
  portals 20
  rdma    20

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    5246b66 View commit details
    Browse the repository at this point in the history

  13. osc/rdma: Fix double assignment typo 

    Fix an assignment of request onto itself, which was a harmless typo,
but was responsible for Coverity issues CID 1429865 and CID 1429864.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
    @bwbarrett
    bwbarrett committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    ad9fae9 View commit details
    Browse the repository at this point in the history