Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MPI_T Events #8057

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

MPI_T Events #8057

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
9fd3e4e
Modify PERUSE code to implement MPI_T Events based on MPI 4.0 proposal.
hjelmn Apr 11, 2018
ed1aead
Modifications for changes to proposed MPI_T Events API as well as min…
cchambreau Mar 23, 2020
1564ea4
remove include of dss.h
hppritcha Nov 24, 2021
45fe4a0
Remove non-conformant event_get_info extent argument and internal tra…
cchambreau Dec 23, 2021
9350bee
Add user_data argument to MPI_T_event_handle_free function.
cchambreau Dec 23, 2021
a0a398e
Fix generation of event_get_info displacements and related copy_data …
cchambreau Dec 24, 2021
4b953c4
Minor cleanup
cchambreau Dec 24, 2021
2152799
mpi_t events: convert to new profiling approach
hppritcha Jan 18, 2022
7d81266
mpi_t events fix some missing files for
hppritcha Jan 18, 2022
50b84fc
pr review feedback
hppritcha Jan 18, 2022
fb8a5f5
pr feedback - switch to opal_clock_gettime
hppritcha Jan 18, 2022
f4ffec2
suppress a compiler warning
hppritcha Jan 18, 2022
6cf3312
pr feedback - remove assert
cchambreau Jun 1, 2022
12dca5d
pr feedback - corrected comment references to events
cchambreau Jun 1, 2022
e6a0389
pr feedback - added locking for setting dropped event handler
cchambreau Jun 1, 2022
7e617d4
pr feedback - copyright consistency for all event files
cchambreau Jun 28, 2022
d91a7ee
Protect source data access with thread locking.
cchambreau Jun 29, 2022
35e202a
Add OPAL_LIKELY
cchambreau Jun 29, 2022
88c7717
Correct OSC event descriptions.
cchambreau Jun 29, 2022
9ac3f5f
Ensure consistency between ompi and opal callback safety requirement
cchambreau Jun 30, 2022
8853c60
Ensure thread safety for incremented variables.
cchambreau Jun 30, 2022
750af0c
Add function to lookup ompi datatype from opal datatype id.
cchambreau Jun 30, 2022
66b29d5
fix compile problem after rebase
hppritcha Jan 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions configure.ac
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,12 @@ AC_MSG_CHECKING([for bootstrap Libtool version])
ltversion=`grep VERSION= $srcdir/config/ltmain.sh | head -n 1 | cut -d= -f2`
AC_MSG_RESULT([$ltversion])

# Ensure mpi.h.in and mca_base_event.h have consistent callback safety values
AC_DEFINE_UNQUOTED([OPAL_MCA_BASE_CB_REQUIRE_NONE], [0], [Keeping OPAL and OMPI values in sync])
AC_DEFINE_UNQUOTED([OPAL_MCA_BASE_CB_REQUIRE_MPI_RESTRICTED], [1], [Keeping OPAL and OMPI values in sync])
AC_DEFINE_UNQUOTED([OPAL_MCA_BASE_CB_REQUIRE_THREAD_SAFE], [2], [Keeping OPAL and OMPI values in sync])
AC_DEFINE_UNQUOTED([OPAL_MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE], [3], [Keeping OPAL and OMPI values in sync])

# List header files to generate

AC_CONFIG_HEADERS([opal/include/opal_config.h])
Expand Down Expand Up @@ -1399,6 +1405,7 @@ if test $ac_cv_header_sys_synch_h = yes ; then
[Do not use outside of mpi.h. Define to 1 if you have the <sys/synch.h> header file.])
fi


# If there is a local hook for each project, call it. This allows 3rd
# parties to add configuration steps to OPAL and/or OMPI simply
# by placing a file in [opal|ompi]/config/whatever.m4 that
Expand Down
1 change: 1 addition & 0 deletions ompi/datatype/Makefile.am
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ libdatatype_la_SOURCES = \
ompi_datatype_create_subarray.c \
ompi_datatype_external.c \
ompi_datatype_external32.c \
ompi_datatype_lookup_by_opal_id.c \
ompi_datatype_match_size.c \
ompi_datatype_module.c \
ompi_datatype_sndrcv.c \
Expand Down
2 changes: 2 additions & 0 deletions ompi/datatype/ompi_datatype.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,8 @@ ompi_datatype_copy_content_same_ddt( const ompi_datatype_t* type, size_t count,
return 0;
}

OMPI_DECLSPEC ompi_datatype_t* ompi_datatype_lookup_by_opal_id( uint16_t opal_id );

OMPI_DECLSPEC const ompi_datatype_t* ompi_datatype_match_size( int size, uint16_t datakind, uint16_t datalang );

/*
Expand Down
28 changes: 28 additions & 0 deletions ompi/datatype/ompi_datatype_lookup_by_opal_id.c
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
/* -*- Mode: C; c-basic-offset:4 ; -*- */
/*
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/

#include "ompi_config.h"
#include "ompi/datatype/ompi_datatype.h"
#include "ompi/datatype/ompi_datatype_internal.h"


ompi_datatype_t* ompi_datatype_lookup_by_opal_id( uint16_t opal_id )
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OPAL and OMPI datatypes are different things, and multiple OMPI datatypes will be mapped unto the same underlying OPAL datatype. Thus, looking for an OMPI datatype that matches a specific OPAL id, will give you the first OMPI datatype that matches that id, and that might be an OMPI datatype that is equivalent to your original one, but not necessarily the same. Why are you not always manipulating OMPI datatype id (which would be unique for each datatype).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bosilca I see what you are saying. The underlying event support is implemented in opal/mca/base/mca_base_event.* with opal_datatype_t, which is a different size than ompi_datatype_t (uint16_t / uint32_t). The currently implemented events are in OMPI modules. Do you have a suggestion for tracking OMPI datatypes with the OPAL support?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the intent here is:

  • OPAL provides a basic datatype engine. It's not specifically tied to the MPI API or MPI semantics. The OPAL datatype engine does generic datatype-like things.
  • The datatype functionality in MPI layer adds further semantics and data structures on top of the OPAL datatype engine to implement the MPI-specific datatype functionality.

I think @bosilca is asking to maintain the clean separation between the two. Hence, you'll need to look up MPI datatypes as MPI datatypes -- not OPAL datatypes.

The same concept applies to the base MPI_T event support: it should be tracking MPI events and MPI datatypes and other MPI objects. If desirable, there could be an underlying OPAL "event tracking" engine that does things in an MPI-agnostic way (wholly independent of the MPI layer APIs and MPI semantics). This would be useful, for example, if we want to be able to track OPAL-level events that have no knowledge of the MPI layer (the BTLs seem like an obvious candidate here, for example).

The MPI_T functionality in the MPI layer should be able to call down into the OPAL "event tracking" engine to track MPI-level events.

The tricky part will then be how to correlate MPI-level events with OPAL-level events (e.g., how to correlate BTL events to specific MPI communicators, for example).

{
int32_t i;
const ompi_datatype_t* datatype = NULL;

for (int j = 0 ; j < OMPI_DATATYPE_MPI_MAX_PREDEFINED ; ++j) {
if (ompi_datatype_basicDatatypes[j]->super.id == opal_id) {
datatype = (ompi_datatype_t *) ompi_datatype_basicDatatypes[j];
break;
}
}

return datatype;
}
117 changes: 116 additions & 1 deletion ompi/include/mpi.h.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@
* Copyright (c) 2021-2022 Amazon.com, Inc. or its affiliates. All Rights
* reserved.
* Copyright (c) 2021 Bull S.A.S. All rights reserved.
* Copyright (c) 2018 Triad National Security, LLC. All rights
* Copyright (c) 2018-2022 Triad National Security, LLC. All rights
* reserved.
* $COPYRIGHT$
Expand Down Expand Up @@ -461,6 +460,8 @@ typedef struct ompi_mpit_cvar_handle_t *MPI_T_cvar_handle;
typedef struct mca_base_pvar_handle_t *MPI_T_pvar_handle;
typedef struct mca_base_pvar_session_t *MPI_T_pvar_session;
typedef struct ompi_instance_t *MPI_Session;
typedef struct mca_base_raised_event_t *MPI_T_event_instance;
typedef struct mca_base_event_registration_t *MPI_T_event_registration;

/*
* MPI_Status
Expand Down Expand Up @@ -920,6 +921,48 @@ enum {
MPI_T_PVAR_CLASS_GENERIC
};

/*
* MPIT callback safety levels
*
* Values are set in configure.ac for consistency with mca_base_event.h
*/
#undef OPAL_MCA_BASE_CB_REQUIRE_NONE
#undef OPAL_MCA_BASE_CB_REQUIRE_MPI_RESTRICTED
#undef OPAL_MCA_BASE_CB_REQUIRE_THREAD_SAFE
#undef OPAL_MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE
typedef enum {
MPI_T_CB_REQUIRE_NONE = OPAL_MCA_BASE_CB_REQUIRE_NONE,
MPI_T_CB_REQUIRE_MPI_RESTRICTED = OPAL_MCA_BASE_CB_REQUIRE_MPI_RESTRICTED,
MPI_T_CB_REQUIRE_THREAD_SAFE = OPAL_MCA_BASE_CB_REQUIRE_THREAD_SAFE,
MPI_T_CB_REQUIRE_ASYNC_SIGNAL_SAFE = OPAL_MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE
} MPI_T_cb_safety;

/*
* MPIT source ordering
*/
enum ompi_mpi_t_source_order_t {
MPI_T_ORDERED,
MPI_T_UNORDERED,
};

typedef enum ompi_mpi_t_source_order_t MPI_T_source_order;

/*
* MPI Tool event functions
*/
typedef void (*MPI_T_event_free_cb_function) (MPI_T_event_registration handle,
MPI_T_cb_safety cb_safety,
void *user_data);
typedef void (*MPI_T_event_dropped_cb_function) (MPI_Count count,
MPI_T_event_registration handle,
int source_index,
MPI_T_cb_safety cb_safety,
void *user_data);
typedef void (*MPI_T_event_cb_function) (MPI_T_event_instance event,
MPI_T_event_registration handle,
MPI_T_cb_safety cb_safety,
void *user_data);

/*
* NULL handles
*/
Expand Down Expand Up @@ -955,6 +998,7 @@ enum {
#define MPI_T_PVAR_HANDLE_NULL ((MPI_T_pvar_handle) 0)
#define MPI_T_PVAR_SESSION_NULL ((MPI_T_pvar_session) 0)
#define MPI_T_CVAR_HANDLE_NULL ((MPI_T_cvar_handle) 0)
#define MPI_T_EVENT_REGISTRATION_NULL ((MPI_T_event_registration) 0)

/* MPI-2 specifies that the name "MPI_TYPE_NULL_DELETE_FN" (and all
related friends) must be accessible in C, C++, and Fortran. This is
Expand Down Expand Up @@ -2953,6 +2997,41 @@ OMPI_DECLSPEC int PMPI_T_enum_get_info(MPI_T_enum enumtype, int *num, char *nam
OMPI_DECLSPEC int PMPI_T_enum_get_item(MPI_T_enum enumtype, int index, int *value, char *name,
int *name_len);

OMPI_DECLSPEC int PMPI_T_event_get_num (int *num_events);
OMPI_DECLSPEC int PMPI_T_event_get_info (int event_index, char *name, int *name_len,
int *verbosity, MPI_Datatype *array_of_datatypes,
MPI_Aint *array_of_displacements, int *num_elements,
MPI_T_enum *enumtype, MPI_Info *info,
char *desc, int *desc_len, int *bind);
OMPI_DECLSPEC int PMPI_T_event_get_index (const char *name, int *event_index);
OMPI_DECLSPEC int PMPI_T_event_handle_alloc (int event_index, void *obj_handle,
MPI_Info info, MPI_T_event_registration *event_registration);
OMPI_DECLSPEC int PMPI_T_event_handle_set_info (MPI_T_event_registration event_registration, MPI_Info info);
OMPI_DECLSPEC int PMPI_T_event_handle_get_info (MPI_T_event_registration event_registration,
MPI_Info *info_used);
OMPI_DECLSPEC int PMPI_T_event_register_callback (MPI_T_event_registration event_registration,
MPI_T_cb_safety cb_safety, MPI_Info info, void *user_data,
MPI_T_event_cb_function event_cb_function);
OMPI_DECLSPEC int PMPI_T_event_callback_set_info (MPI_T_event_registration event_registration,
MPI_T_cb_safety cb_safety, MPI_Info info);
OMPI_DECLSPEC int PMPI_T_event_callback_get_info (MPI_T_event_registration event_registration,
MPI_T_cb_safety cb_safety, MPI_Info *info_used);
OMPI_DECLSPEC int PMPI_T_event_handle_free (MPI_T_event_registration event_registration,
void *user_data,
MPI_T_event_free_cb_function free_cb_function);
OMPI_DECLSPEC int PMPI_T_event_set_dropped_handler (MPI_T_event_registration handle,
MPI_T_event_dropped_cb_function dropped_cb_function);
OMPI_DECLSPEC int PMPI_T_event_read (MPI_T_event_instance event, int element_index, void *buffer);
OMPI_DECLSPEC int PMPI_T_event_copy (MPI_T_event_instance event, void *buffer);
OMPI_DECLSPEC int PMPI_T_event_get_timestamp (MPI_T_event_instance event, MPI_Count *event_time);
OMPI_DECLSPEC int PMPI_T_event_get_source (MPI_T_event_instance event, int *source_index);
OMPI_DECLSPEC int PMPI_T_source_get_num (int *num_source);
OMPI_DECLSPEC int PMPI_T_source_get_info (int source_id, char *name, int *name_len,
char *desc, int *desc_len, MPI_T_source_order *ordering,
MPI_Count *ticks_per_second, MPI_Count *max_timestamp,
MPI_Info *info);
OMPI_DECLSPEC int PMPI_T_source_get_timestamp (int source_id, MPI_Count *timestamp);

/*
* Tool MPI API
*/
Expand Down Expand Up @@ -3032,6 +3111,42 @@ OMPI_DECLSPEC int PMPI_Info_get_valuelen(MPI_Info info, const char *key, int *v
int *flag)
__mpi_interface_deprecated_in_mpi40__("PMPI_Info_get_valuelen was deprecated in MPI-4.0; use PMPI_Info_get_string instead");

OMPI_DECLSPEC int MPI_T_event_get_num (int *num_events);
OMPI_DECLSPEC int MPI_T_event_get_info (int event_index, char *name, int *name_len,
int *verbosity, MPI_Datatype *array_of_datatypes,
MPI_Aint *array_of_displacements, int *num_elements,
MPI_T_enum *enumtype, MPI_Info *info,
char *desc, int *desc_len, int *bind);
OMPI_DECLSPEC int MPI_T_event_get_index (const char *name, int *event_index);
OMPI_DECLSPEC int MPI_T_event_handle_alloc (int event_index, void *obj_handle,
MPI_Info info, MPI_T_event_registration *event_registration);
OMPI_DECLSPEC int MPI_T_event_handle_set_info (MPI_T_event_registration event_registration, MPI_Info info);
OMPI_DECLSPEC int MPI_T_event_handle_get_info (MPI_T_event_registration event_registration,
MPI_Info *info_used);
OMPI_DECLSPEC int MPI_T_event_handle_free (MPI_T_event_registration event_registration,
void *user_data,
MPI_T_event_free_cb_function free_cb_function);
OMPI_DECLSPEC int MPI_T_event_register_callback (MPI_T_event_registration event_registration,
MPI_T_cb_safety cb_safety, MPI_Info info, void *user_data,
MPI_T_event_cb_function event_cb_function);
OMPI_DECLSPEC int MPI_T_event_callback_set_info (MPI_T_event_registration event_registration,
MPI_T_cb_safety cb_safety, MPI_Info info);
OMPI_DECLSPEC int MPI_T_event_callback_get_info (MPI_T_event_registration event_registration,
MPI_T_cb_safety cb_safety, MPI_Info *info_used);
OMPI_DECLSPEC int MPI_T_event_set_dropped_handler (MPI_T_event_registration handle,
MPI_T_event_dropped_cb_function dropped_cb_function);
OMPI_DECLSPEC int MPI_T_event_read (MPI_T_event_instance event, int element_index, void *buffer);
OMPI_DECLSPEC int MPI_T_event_copy (MPI_T_event_instance event, void *buffer);
OMPI_DECLSPEC int MPI_T_event_get_timestamp (MPI_T_event_instance event, MPI_Count *event_time);
OMPI_DECLSPEC int MPI_T_event_get_source (MPI_T_event_instance event, int *source_index);

OMPI_DECLSPEC int MPI_T_source_get_num (int *num_source);
OMPI_DECLSPEC int MPI_T_source_get_info (int source_id, char *name, int *name_len,
char *desc, int *desc_len, MPI_T_source_order *ordering,
MPI_Count *ticks_per_second, MPI_Count *max_timestamp,
MPI_Info *info);
OMPI_DECLSPEC int MPI_T_source_get_timestamp (int source_id, MPI_Count *timestamp);

/*
* Even though MPI_Copy_function and MPI_Delete_function are
* deprecated, we do not use the attributes marking them as such,
Expand Down
28 changes: 28 additions & 0 deletions ompi/mca/osc/rdma/osc_rdma.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,34 @@
#include "osc_rdma_peer.h"

#include "opal_stdint.h"
#include "opal/mca/base/mca_base_event.h"

enum {
OMPI_OSC_RDMA_EVENT_LOCK_ACQUIRED,
OMPI_OSC_RDMA_EVENT_LOCK_RELEASED,
OMPI_OSC_RDMA_EVENT_PUT_STARTED,
OMPI_OSC_RDMA_EVENT_PUT_COMPLETE,
OMPI_OSC_RDMA_EVENT_GET_STARTED,
OMPI_OSC_RDMA_EVENT_GET_COMPLETE,
OMPI_OSC_RDMA_EVENT_FLUSH_STARTED,
OMPI_OSC_RDMA_EVENT_FLUSH_COMPLETE,
OMPI_OSC_RDMA_EVENT_PSCW_EXPOSE_START,
OMPI_OSC_RDMA_EVENT_PSCW_EXPOSE_COMPLETE,
OMPI_OSC_RDMA_EVENT_PSCW_ACCESS_START,
OMPI_OSC_RDMA_EVENT_PSCW_ACCESS_COMPLETE,
OMPI_OSC_RDMA_EVENT_FENCE,
OMPI_OSC_RDMA_EVENT_MAX,
};

struct mca_osc_rdma_rdma_event_t {
int target;
uint64_t address;
uint64_t size;
};

typedef struct mca_osc_rdma_rdma_event_t mca_osc_rdma_rdma_event_t;

extern mca_base_event_list_item_t mca_osc_rdma_events[];

#define RANK_ARRAY_COUNT(module) ((ompi_comm_size ((module)->comm) + (module)->node_count - 1) / (module)->node_count)

Expand Down
16 changes: 16 additions & 0 deletions ompi/mca/osc/rdma/osc_rdma_active_target.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,8 @@ static void ompi_osc_rdma_handle_post (ompi_osc_rdma_module_t *module, int rank,
rank, (int) (npeers - state->num_post_msgs - 1));
/* an atomic is not really necessary as this function is currently used but it doesn't hurt */
ompi_osc_rdma_counter_add (&state->num_post_msgs, 1);
MCA_BASE_EVENT_RAISE(mca_osc_rdma_events[OMPI_OSC_RDMA_EVENT_PSCW_ACCESS_START].event, MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE,
module->win, NULL, &rank);
return;
}
}
Expand Down Expand Up @@ -307,6 +309,9 @@ int ompi_osc_rdma_post_atomic (ompi_group_t *group, int mpi_assert, ompi_win_t *
return OMPI_SUCCESS;
}

MCA_BASE_EVENT_RAISE(mca_osc_rdma_events[OMPI_OSC_RDMA_EVENT_PSCW_EXPOSE_START].event, MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE,
module->win, NULL, NULL);

/* translate group ranks into the communicator */
peers = ompi_osc_rdma_get_peers (module, module->pw_group);
if (OPAL_UNLIKELY(NULL == peers)) {
Expand Down Expand Up @@ -394,6 +399,8 @@ int ompi_osc_rdma_start_atomic (ompi_group_t *group, int mpi_assert, ompi_win_t
"from %d processes", peer->rank, (int) (group_size - state->num_post_msgs - 1));
opal_list_remove_item (&module->pending_posts, &pending_post->super);
OBJ_RELEASE(pending_post);
MCA_BASE_EVENT_RAISE(mca_osc_rdma_events[OMPI_OSC_RDMA_EVENT_PSCW_ACCESS_START].event, MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE,
module->win, NULL, &peer->rank);
ompi_osc_rdma_counter_add (&state->num_post_msgs, 1);
break;
}
Expand Down Expand Up @@ -464,6 +471,9 @@ int ompi_osc_rdma_complete_atomic (ompi_win_t *win)
ompi_osc_rdma_peer_t *peer = peers[i];
intptr_t target = (intptr_t) peer->state + offsetof (ompi_osc_rdma_state_t, num_complete_msgs);

MCA_BASE_EVENT_RAISE(mca_osc_rdma_events[OMPI_OSC_RDMA_EVENT_PSCW_ACCESS_COMPLETE].event, MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE,
module->win, NULL, &peer->rank);

if (!ompi_osc_rdma_peer_local_state (peer)) {
ret = ompi_osc_rdma_lock_btl_op (module, peer, target, MCA_BTL_ATOMIC_ADD, 1, true);
assert (OMPI_SUCCESS == ret);
Expand Down Expand Up @@ -507,6 +517,9 @@ int ompi_osc_rdma_wait_atomic (ompi_win_t *win)
opal_atomic_mb ();
}

MCA_BASE_EVENT_RAISE(mca_osc_rdma_events[OMPI_OSC_RDMA_EVENT_PSCW_EXPOSE_COMPLETE].event, MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE,
module->win, NULL, NULL);

OPAL_THREAD_LOCK(&module->lock);
group = module->pw_group;
module->pw_group = NULL;
Expand Down Expand Up @@ -556,6 +569,9 @@ int ompi_osc_rdma_test_atomic (ompi_win_t *win, int *flag)
module->pw_group = NULL;
OPAL_THREAD_UNLOCK(&(module->lock));

MCA_BASE_EVENT_RAISE(mca_osc_rdma_events[OMPI_OSC_RDMA_EVENT_PSCW_EXPOSE_COMPLETE].event, MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE,
module->win, NULL, NULL);

OBJ_RELEASE(group);

OSC_RDMA_VERBOSE(MCA_BASE_VERBOSE_TRACE, "test complete. returning flag: true");
Expand Down
9 changes: 7 additions & 2 deletions ompi/mca/osc/rdma/osc_rdma_comm.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@
* $HEADER$
*/

#include "ompi_config.h"

#include "osc_rdma.h"
#include "osc_rdma_comm.h"
#include "osc_rdma_frag.h"
#include "osc_rdma_sync.h"
Expand Down Expand Up @@ -461,6 +460,9 @@ static int ompi_osc_rdma_put_real (ompi_osc_rdma_sync_t *sync, ompi_osc_rdma_pee
OSC_RDMA_VERBOSE(MCA_BASE_VERBOSE_TRACE, "initiating btl put of %lu bytes to remote address %" PRIx64 ", sync "
"object %p...", (unsigned long) size, target_address, (void *) sync);

MCA_BASE_EVENT_RAISE(mca_osc_rdma_events[OMPI_OSC_RDMA_EVENT_PUT_STARTED].event, MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE,
module->win, NULL, &((mca_osc_rdma_rdma_event_t){.target = peer->rank, .address = target_address, .size = size}));

/* flag outstanding rma requests */
ompi_osc_rdma_sync_rdma_inc (sync);

Expand Down Expand Up @@ -725,6 +727,9 @@ static int ompi_osc_rdma_get_contig (ompi_osc_rdma_sync_t *sync, ompi_osc_rdma_p
ompi_osc_rdma_sync_rdma_inc (sync);
}

MCA_BASE_EVENT_RAISE(mca_osc_rdma_events[OMPI_OSC_RDMA_EVENT_GET_STARTED].event, MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE,
module->win, NULL, &((mca_osc_rdma_rdma_event_t){.target = peer->rank, .address = source_address, .size = size}));

do {
ret = ompi_osc_rdma_btl_get(module, peer->data_btl_index, peer->data_endpoint,
ptr, aligned_source_base, local_handle, source_handle,
Expand Down
Loading