Updated ABI generation code and new libraries #13280
Conversation
hppritcha
force-pushed
the
abi-generate-ver2
branch
2 times, most recently
from
8fbf968 to
29b1a2f
Compare
hppritcha
force-pushed
the
abi-generate-ver2
branch
from
da515d4 to
340e7ad
Compare
|
Maybe you should somehow vendor the In short, I think it would be in everyone's convenience to use https://github.com/mpi-forum/mpi-abi-stubs as the "source of truth" for ABI-related stuff, avoiding manual synchronization of handle/constant values. |
|
Hello! The Git Commit Checker CI bot found a few problems with this PR: f7d94fa: WIP: explain issue with pympistandard for callback...
4d79937: WIP: fix JSONs
ec9c45a: WIP: fix typo in pympistd arg
b047b19: WIP: add JSONs for ABI and API
c9d0c7a: WIP: bump pympistandard commit for profiling embig...
a1255ce: WIP: move Aint helper macros under ifndef OMPI_NO_...
342b072: WIP: add some workarounds for MPI_Fint and MPI_Inf...
53aeac5: WIP: mangle some more functions
93c0a39: WIP: avoid double inclusion of abi.h
afd9eb2: WIP: use pympistandard by editing PYTHONPATH (inst...
d397bfc: WIP: fix some bugs in mangling names
da2630f: WIP: fix typo for MPI_internal
956dded: WIP: add additional types and functions to be mang...
0b10ce6: WIP: temp fix for Aint problems
9830327: WIP: add input for abi.h.in
68c69df: WIP: move abi.h.in
702cb23: WIP: add in 5.0 apis.json
6b2784f: WIP: move code out of consts.py
22d4cb2: WIP: call c_header from Makefile
b0d0ff2: WIP: mangle names for internal usage
96a33c3: WIP: generate callback function prototypes
d2dd7ed: WIP: remove comment function
45d8789: WIP: print out embiggened versions of functions
b081aa2: WIP: add MPI and ABI versions
80ebfe7: WIP: generate API prototypes
93e6375: WIP: Comment out a Fortran-only category
db8a2b5: WIP: add comment pointing back to MPI standard
08dfb42: WIP: use enums for most int values
aab2023: WIP: create ABI header file from template with cat...
46fae8e: WIP: generate header with ABI values for #defines
c15a05d: WIP: remove abi.py
Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks! |
1 similar comment
|
Hello! The Git Commit Checker CI bot found a few problems with this PR: f7d94fa: WIP: explain issue with pympistandard for callback...
4d79937: WIP: fix JSONs
ec9c45a: WIP: fix typo in pympistd arg
b047b19: WIP: add JSONs for ABI and API
c9d0c7a: WIP: bump pympistandard commit for profiling embig...
a1255ce: WIP: move Aint helper macros under ifndef OMPI_NO_...
342b072: WIP: add some workarounds for MPI_Fint and MPI_Inf...
53aeac5: WIP: mangle some more functions
93c0a39: WIP: avoid double inclusion of abi.h
afd9eb2: WIP: use pympistandard by editing PYTHONPATH (inst...
d397bfc: WIP: fix some bugs in mangling names
da2630f: WIP: fix typo for MPI_internal
956dded: WIP: add additional types and functions to be mang...
0b10ce6: WIP: temp fix for Aint problems
9830327: WIP: add input for abi.h.in
68c69df: WIP: move abi.h.in
702cb23: WIP: add in 5.0 apis.json
6b2784f: WIP: move code out of consts.py
22d4cb2: WIP: call c_header from Makefile
b0d0ff2: WIP: mangle names for internal usage
96a33c3: WIP: generate callback function prototypes
d2dd7ed: WIP: remove comment function
45d8789: WIP: print out embiggened versions of functions
b081aa2: WIP: add MPI and ABI versions
80ebfe7: WIP: generate API prototypes
93e6375: WIP: Comment out a Fortran-only category
db8a2b5: WIP: add comment pointing back to MPI standard
08dfb42: WIP: use enums for most int values
aab2023: WIP: create ABI header file from template with cat...
46fae8e: WIP: generate header with ABI values for #defines
c15a05d: WIP: remove abi.py
Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks! |
hppritcha
force-pushed
the
abi-generate-ver2
branch
from
a9b105c to
cefbde8
Compare
|
Hello! The Git Commit Checker CI bot found a few problems with this PR: e86186b: WIP: add JSONs for ABI and API
24417ec: WIP: bump pympistandard commit for profiling embig...
718b1d0: WIP: move Aint helper macros under ifndef OMPI_NO_...
3431c8d: WIP: add some workarounds for MPI_Fint and MPI_Inf...
566fdaa: WIP: mangle some more functions
8396bef: WIP: avoid double inclusion of abi.h
39c20b7: WIP: use pympistandard by editing PYTHONPATH (inst...
d1aece4: WIP: fix some bugs in mangling names
c7c1809: WIP: fix typo for MPI_internal
2bbf9eb: WIP: add additional types and functions to be mang...
1db8082: WIP: temp fix for Aint problems
870e925: WIP: add input for abi.h.in
a56da85: WIP: move abi.h.in
4c2aee1: WIP: add in 5.0 apis.json
3d85943: WIP: move code out of consts.py
4e85726: WIP: call c_header from Makefile
8d8f554: WIP: mangle names for internal usage
5f29a48: WIP: generate callback function prototypes
c5d5f30: WIP: remove comment function
1613149: WIP: print out embiggened versions of functions
3f229b3: WIP: add MPI and ABI versions
584aeb7: WIP: generate API prototypes
7c73091: WIP: Comment out a Fortran-only category
72d2bff: WIP: add comment pointing back to MPI standard
c303345: WIP: use enums for most int values
7d79681: WIP: create ABI header file from template with cat...
99e9992: WIP: generate header with ABI values for #defines
Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks! |
|
I wonder if we should make the bot not complain about unsigned commits on draft PRs. That would reduce some of the noise on PR's like this. |
hppritcha
force-pushed
the
abi-generate-ver2
branch
from
cefbde8 to
d5663f7
Compare
Turns out that in commit 6bd36a7 we had a function that is not part of the MPI standard. This showed while working on ABI support - which requires us to pay attention to the truth rather than make stuff up. This commit removes our made up MPI_Session_set_info method. Turns out who ever was doing the fortran bindings knew this wasn't a method in the standard so there's no need to change the fortran bindings. Same thing applies to the man pages. Related to open-mpi#13280 Signed-off-by: Howard Pritchard <howardp@lanl.gov>
hppritcha
force-pushed
the
abi-generate-ver2
branch
from
65e34c9 to
bd2710f
Compare
|
@hppritcha There is some issue with out-of-source builds, i.e |
|
interesting distcheck didn't check this. |
|
jenkins ci runs make distcheck |
|
The ABI mpi.h header is missing some MPI_T_XXX types. Also, the Status f08/c converters are declared, and they should not. I did the following manual edits to the installed mpi.h header: diff -up ./mpi.h.orig ./mpi.h
--- ./mpi.h.orig 2025-08-28 18:55:49.842968779 +0300
+++ ./mpi.h 2025-08-28 19:03:07.192957305 +0300
@@ -490,6 +490,13 @@ enum {
/* C preprocessor constants and Fortran parameters */
/* $CATEGORY:C_PREPROCESSOR_CONSTANTS_FORTRAN_PARAMETERS$ */
+typedef struct MPI_T_enum_t* MPI_T_enum;
+typedef struct MPI_T_cvar_handle_t* MPI_T_cvar_handle;
+typedef struct MPI_T_pvar_handle_t* MPI_T_pvar_handle;
+typedef struct MPI_T_pvar_session_t* MPI_T_pvar_session;
+typedef struct MPI_T_event_registration_t* MPI_T_event_registration;
+typedef struct MPI_T_event_instance_t* MPI_T_event_instance;
+
/* Handles used in the MPI tool information interface */
#define MPI_T_ENUM_NULL ((MPI_T_enum) 0)
#define MPI_T_CVAR_HANDLE_NULL ((MPI_T_cvar_handle) 0)
@@ -558,20 +565,20 @@ enum {
};
/* Source event ordering guarantees in the MPI tool information interface */
-enum {
+typedef enum MPI_T_source_order {
MPI_T_SOURCE_ORDERED = 1,
MPI_T_SOURCE_UNORDERED = 2,
-};
+} MPI_T_source_order;
/*
* Callback safety requirement levels used in the MPI tool information interface
*/
-enum {
+typedef enum MPI_T_cb_safety {
MPI_T_CB_REQUIRE_NONE = 0x00,
MPI_T_CB_REQUIRE_MPI_RESTRICTED = 0x03,
MPI_T_CB_REQUIRE_THREAD_SAFE = 0x0F,
MPI_T_CB_REQUIRE_ASYNC_SIGNAL_SAFE = 0x3F,
-};
+} MPI_T_cb_safety;
/* Callback functions */
@@ -1107,13 +1114,13 @@ int MPI_Ssend_init(const void* buf, int
int MPI_Ssend_init_c(const void* buf, MPI_Count count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request* request);
int MPI_Start(MPI_Request* request);
int MPI_Startall(int count, MPI_Request array_of_requests[]);
-/* int MPI_Status_c2f(const MPI_Status* c_status, MPI_Fint* f_status);
- */int MPI_Status_c2f08(const MPI_Status* c_status, MPI_F08_status* f08_status);
-int MPI_Status_f082c(const MPI_F08_status* f08_status, MPI_Status* c_status);
-/* int MPI_Status_f082f(const MPI_F08_status* f08_status, MPI_Fint* f_status);
- *//* int MPI_Status_f2c(const MPI_Fint* f_status, MPI_Status* c_status);
- *//* int MPI_Status_f2f08(const MPI_Fint* f_status, MPI_F08_status* f08_status);
- */int MPI_Status_get_error(const MPI_Status* status, int* err);
+// /* int MPI_Status_c2f(const MPI_Status* c_status, MPI_Fint* f_status);
+// */int MPI_Status_c2f08(const MPI_Status* c_status, MPI_F08_status* f08_status);
+// int MPI_Status_f082c(const MPI_F08_status* f08_status, MPI_Status* c_status);
+// /* int MPI_Status_f082f(const MPI_F08_status* f08_status, MPI_Fint* f_status);
+// *//* int MPI_Status_f2c(const MPI_Fint* f_status, MPI_Status* c_status);
+// *//* int MPI_Status_f2f08(const MPI_Fint* f_status, MPI_F08_status* f08_status);
+// */int MPI_Status_get_error(const MPI_Status* status, int* err);
int MPI_Status_get_source(const MPI_Status* status, int* source);
int MPI_Status_get_tag(const MPI_Status* status, int* tag);
int MPI_Status_set_cancelled(MPI_Status* status, int flag);
@@ -1799,13 +1806,13 @@ int PMPI_Ssend_init(const void* buf, int
int PMPI_Ssend_init_c(const void* buf, MPI_Count count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request* request);
int PMPI_Start(MPI_Request* request);
int PMPI_Startall(int count, MPI_Request array_of_requests[]);
-/* int PMPI_Status_c2f(const MPI_Status* c_status, MPI_Fint* f_status);
- */int PMPI_Status_c2f08(const MPI_Status* c_status, MPI_F08_status* f08_status);
-int PMPI_Status_f082c(const MPI_F08_status* f08_status, MPI_Status* c_status);
-/* int PMPI_Status_f082f(const MPI_F08_status* f08_status, MPI_Fint* f_status);
- *//* int PMPI_Status_f2c(const MPI_Fint* f_status, MPI_Status* c_status);
- *//* int PMPI_Status_f2f08(const MPI_Fint* f_status, MPI_F08_status* f08_status);
- */int PMPI_Status_get_error(const MPI_Status* status, int* err);
+// /* int PMPI_Status_c2f(const MPI_Status* c_status, MPI_Fint* f_status);
+// */int PMPI_Status_c2f08(const MPI_Status* c_status, MPI_F08_status* f08_status);
+// int PMPI_Status_f082c(const MPI_F08_status* f08_status, MPI_Status* c_status);
+// /* int PMPI_Status_f082f(const MPI_F08_status* f08_status, MPI_Fint* f_status);
+// *//* int PMPI_Status_f2c(const MPI_Fint* f_status, MPI_Status* c_status);
+// *//* int PMPI_Status_f2f08(const MPI_Fint* f_status, MPI_F08_status* f08_status);
+// */int PMPI_Status_get_error(const MPI_Status* status, int* err);
int PMPI_Status_get_source(const MPI_Status* status, int* source);
int PMPI_Status_get_tag(const MPI_Status* status, int* tag);
int PMPI_Status_set_cancelled(MPI_Status* status, int flag);I'm able to compile (using |
|
@hppritcha Actually, take a look at mpi-forum/mpi-abi-stubs#63 |
hppritcha
force-pushed
the
abi-generate-ver2
branch
from
5377de6 to
04ff2ff
Compare
|
The installed $ mpicc_abi helloworld.c
$ mpiexec -n 1 ./a.out
[optiplex:00000] *** An error occurred in MPI_Init_thread
[optiplex:00000] *** reported by process [3633774593,0]
[optiplex:00000] *** on a NULL communicator
[optiplex:00000] *** MPI_ERR_ARG: invalid argument of some other kind
[optiplex:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[optiplex:00000] *** and MPI will try to terminate your MPI job as well)
--------------------------------------------------------------------------
prterun has exited due to process rank 0 with PID 0 on node optiplex calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).
-------------------------------------------------------------------------- |
|
@hppritcha The following definitions in the generated /* Predefined functions */
#define MPI_COMM_NULL_COPY_FN ((MPI_Comm_copy_attr_function) 0)
#define MPI_COMM_DUP_FN ((MPI_Comm_copy_attr_function) 1)
#define MPI_COMM_NULL_DELETE_FN ((MPI_Comm_delete_attr_function) 0)
#define MPI_WIN_NULL_COPY_FN ((MPI_Win_copy_attr_function) 0)
#define MPI_WIN_DUP_FN ((MPI_Win_copy_attr_function) 1)
#define MPI_WIN_NULL_DELETE_FN ((MPI_Win_delete_attr_function) 0)
#define MPI_TYPE_NULL_COPY_FN ((MPI_Type_copy_attr_function) 0)
#define MPI_TYPE_DUP_FN ((MPI_Type_copy_attr_function) 1)
#define MPI_TYPE_NULL_DELETE_FN ((MPI_Type_delete_attr_function) 0)
#define MPI_CONVERSION_FN_NULL ((MPI_Datarep_conversion_function) 0)
#define MPI_CONVERSION_FN_NULL_C ((MPI_Datarep_conversion_function_c) 0)
/* Deprecated predefined functions */
#define MPI_NULL_COPY_FN ((MPI_Copy_function) 0)
#define MPI_DUP_FN ((MPI_Copy_function) 1)
#define MPI_NULL_DELETE_FN ((MPI_Delete_function) 0) |
|
PS: You may have a similar problems in some |
ompi/mpi/c/abi_converters.h
Outdated
|
|
||
| __opal_attribute_always_inline__ static inline int ompi_convert_abi_ts_level_intern_ts_level(int ts_level) | ||
| { | ||
| if (MPI_THREAD_SINGLE_ABI_INTERNAL == ts_level) { |
There was a problem hiding this comment.
Wouldn't a switch statement be better? It may ultimately be a matter of taste (optimizing compilers may generate similar binary code). Just a mild observation.
|
@hppritcha What's your plan for stuff introduced in MPI 4.1 and MPI 5.0? The generated mpi.h header says This is a list of what I managed to detect as missing from mpi4py's configuration machinery for missing MPI stuff. |
|
Current status regarding mpi4py:
|
|
Thanks for checking this stuff out in its early state @dalcinl. some of the above functions have been implemented but are sitting in various states in PRs. some of these should be defined - like We still need to add some plumbing in the ompi internals for callbacks. that will come in as part of this PR. The NERSC folks would really like ABI working for Doudna so to the extent there's a "plan" it would be nice to get this working sooner than later. |
needs to go through translation Signed-off-by: Howard Pritchard <howardp@lanl.gov>
from dalcinl Signed-off-by: Howard Pritchard <howardp@lanl.gov>
hppritcha
force-pushed
the
abi-generate-ver2
branch
from
aa4a691 to
1688d76
Compare
|
Hmmm. I don't understand this test: in particular the checks in lines 470-472. I''m not seeing anything in section 3.8.1 about the status fields in the case of flag being false and source being MPI_ANY_SOURCE tag MPI_ANY_TAG. |
eventually we'll use separate SOURCE and DEST (or something like that) types. Signed-off-by: Howard Pritchard <howardp@lanl.gov>
I coded that test long ago, but I think my intention was to test that |
|
okay I see in that case the test makes sense. |
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
|
okay with the current set of commits I can get mpi4py at fc5be43c passing for singleton (VERSION set to MPI 5.1). I'm seeing problems with UCX PML for some cases for MPI_mprobe/improbe so have been using the OB1 PML for testing. the UCX problems don't seem to be related to ABI support. |
|
@hppritcha I'm still getting a bunch of warnings all like the following: I guess you should skip |
I don't think mpi4py is tested regularly with the UCX PML. IIRC, the GitHub Actions workflow does not build with UCX, so all testing happens exclusively with OB1, right? |
|
I'm still seeing this failure, related to the previous status->MPI_ERROR drama. $ mpiexec --oversubscribe -n 1 python test/main.py -q test_request
======================================================================
FAIL: testGetStatusAll (test_request.TestRequestArray.testGetStatusAll)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/dalcinl/Devel/mpi4py/test/test_request.py", line 149, in testGetStatusAll
self.assertEqual(status.error, MPI.SUCCESS)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 16660 != 0
----------------------------------------------------------------------
Ran 12 tests in 0.001s
FAILED (failures=1)Besides the status->MPI_ERROR drama, I suspect there is something else going on there. |
|
@hppritcha Look at the attached patch: ompi-status-out.patch EDIT: I deleted a previous comment and refreshed the patch, there was a missing cast for malloc. You may not like it at first, but IMHO, you an OUT status has to be really treated as INOUT, thus they should be converted on input. Otherwise, it is very hard to ensure the preservation of their contents. For example, recalling the status->MPI_ERROR drama, things should behave as if Open MPI had not touched the MPI_ERROR field, thus you must preserve its value on input. Additionally, imagine that some MPI routine taking an OUT status argument fails with error (e.g because of some bad argument), then the most likely expected behavior is that the contents of the status object/array do not change (although I don't think this is explicitly mandated in the standard). Then, once again, the safe thing to to is to copy ABI->OMPI on input, such that changes to status effectively done in the MPI call are visible, and what MPI does not touch it is preserved. |
|
@hppritcha Could you generate an You can use this new malloc wrapper anywhere required to allocate tmp stuff, with two benefits: a) avoid the malloc(0) warnings in debug builds, b) avoid the usual malloc problem if count is negative but gets converted to unsigned |
|
all good ideas. i'll work on these next week. |
per dalcinl suggestion Signed-off-by: Howard Pritchard <howardp@lanl.gov>
|
hmm the ompi-status patch does not apply cleanly for me but I see what you want to do. |
| code.append('nb_request->data.release_arrays[idx] = NULL;') | ||
| code.append('} else {') | ||
| code.append(f'free({self.tmpname});') | ||
| code.append(f'if (NULL != {self.tmpname}) free({self.tmpname});') |
There was a problem hiding this comment.
free(NULL) is legit, though you may want to keep the null check to skip free() for clarity. However, if you add it here, you should add it everywhere elsewhere.
Untested... diff --git a/ompi/mpi/bindings/ompi_bindings/c_type.py b/ompi/mpi/bindings/ompi_bindings/c_type.py
index 27bad69a35..2a27055e68 100644
--- a/ompi/mpi/bindings/ompi_bindings/c_type.py
+++ b/ompi/mpi/bindings/ompi_bindings/c_type.py
@@ -1329,9 +1329,15 @@ class TypeStausOutStandard(StandardABIType):
code.append(f'MPI_Status *{self.tmpname} = NULL;')
code.append(self.if_should_set_status())
if self.count_param is not None:
- code.append(f'{self.tmpname} = ompi_abi_malloc({self.count_param}, sizeof(MPI_Status));')
+ code.append(f'{self.tmpname} = (MPI_Status *)ompi_abi_malloc({self.count_param}, sizeof(MPI_Status));')
+ code.extend([
+ 'for (int i = 0; i < %s; ++i) {' % (self.count_param,),
+ f'{ConvertFuncs.STATUS}(&{self.tmpname}[i], &{self.name}[i]);',
+ '}',
+ ])
code.append(f'{self.status_argument} = {self.tmpname};')
else:
+ code.append(f'{ConvertFuncs.STATUS}(&{self.tmpname}, {self.name});')
code.append(f'{self.status_argument} = &{self.tmpname};')
code.append('} else {')
if self.count_param is not None: |
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
|
i think i captured that thanks! |
there are many functions where a status is conditionally returned. Trying to treat each of these would be too complex so instead copy contents of the status supplied as input into any temporary status variables then copy back out. Signed-off-by: Howard Pritchard <howardp@lanl.gov>
|
@hppritcha My previous advise on the mallow wrapper did not take into account how IMHO, the only fix (without touching ompi internals) is the following ugly but trivial patch: diff --git a/ompi/mpi/bindings/ompi_bindings/c.py b/ompi/mpi/bindings/ompi_bindings/c.py
index 0cdf0baa6f..3d09708c0c 100644
--- a/ompi/mpi/bindings/ompi_bindings/c.py
+++ b/ompi/mpi/bindings/ompi_bindings/c.py
@@ -702,10 +702,11 @@ extern "C" {
self.dump('{')
lines = []
lines.append('void *aptr = NULL;')
+ lines.append('if (0 == count) count = 1;')
lines.append('if (count > 0) {')
lines.append('aptr = (void *)malloc(count * elsize);')
- lines.append('assert(NULL != aptr);')
lines.append('}')
+ lines.append('assert(NULL != aptr);')
lines.append('return aptr;')
lines.append('}')
lines.append('\n')Maybe you could also consider |
|
|
|
actually the way persistent requests are handled alltoallw_init is NBC in terms of this release mechanism - or at least its supposed to be. |
|
Something is off, valgrind is not clean I tried bumping the following to 9, but it did make valgrind happy |
|
i'll look in to that. I think the problem is the nbc req is not pointing to the right memory location. |
|
BTW, my previous patch from malloc is still flawed. The Here you have a reproducer, which segfaults at mpiexec -n 2 python test/main.py -v test_cco_pr_vec.TestCCOVecInplaceSelf.testAlltoallw2Running the same thing, this time with only one process, and under valgrind mpiexec -n 1 valgrind python test/main.py -v test_cco_pr_vec.TestCCOVecInplaceSelf.testAlltoallw2This generates very noisy output from the Python runtime, but eventually I can spot the following: Regarding the int idx = 0;
...
nb_request->data.release_arrays[idx++] = (void *)sendtypes_tmp;
nb_request->data.release_arrays[idx] = NULL;Are you absolutely sure the OMPI internals have not used this same mechanism already on the returned request? Otherwise, this code will remove any previous buffers registered for deallocation. BTW, in the case of NBC, IIRC int idx = 0;
...
if (NULL != sendtypes_tmp) nb_request->data.release_arrays[idx++] = (void *)sendtypes_tmp;
nb_request->data.release_arrays[idx] = NULL;PS: Sorry, in my attempt to be helpful I'm being too noisy. |
7 participants
Two external MPI libraries are now created: libmpi.so and libmpi_abi.so.
Backend code that was originally in libmpi.la has been extracted into
libopen-mpi.la to be linked into both libraries.
Parts of the Open MPI C interface are now being generated by a python
script (abi.py) from modified source files (named with *.in). This
script generates files for both the ompi ABI and the standard ABI from
the same source file, also including new bigcount interfaces.
To compile standard ABI code, there's a new mpicc_abi compiler wrapper.
The standard ABI does not yet include all functions or symbols, so more
complicated source files will not compile. ROMIO must be disabled for
the code to link, since it's relying on the external MPI interface.
Many todos left:
enable-mca-dsoThis PR supercedes #12033