build: Default to building MCA components in the library

Default to building MCA components into the library, rather than
as dso objects, since most users are not taking advantage of
per-library packaging, and this results in less stat/open/etc. calls
during application startup.

Clean up the static/dso selection logic for componets at the same time.
Before, it was relatively hard to describe what would happen if
there was a mix of --enable-mca-dso and --enable-mca-static arguments.
Now, the code will first look for a component-specific argument, then
a framework-level argument, then a global argument.  If there is a tie,
static is preferred.

Update the NEWS and README.md to document the changes.  Jeff Squyres
wrote the original update to the README file, which was adopted to
the new README.md format.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>

NEWS

@@ -20,7 +20,7 @@ Copyright (c) 2012      Sandia National Laboratories.  All rights reserved.
 Copyright (c) 2012      University of Houston. All rights reserved.
 Copyright (c) 2013      NVIDIA Corporation.  All rights reserved.
 Copyright (c) 2013-2018 Intel, Inc. All rights reserved.
-Copyright (c) 2018      Amazon.com, Inc. or its affiliates.  All Rights
+Copyright (c) 2018-2021 Amazon.com, Inc. or its affiliates.  All Rights
                         reserved.
 $COPYRIGHT$
 
@@ -79,6 +79,8 @@ Master (not on release branches yet)
 - Do not build Open SHMEM layer when there are no SPMLs available.
   Currently, this means the Open SHMEM layer will only build if
   a MXM or UCX library is found.
+- Change the default component build behavior to prefer building
+  components as part of libmpi.so instead of individual DSOs.
 
 4.0.5 -- August, 2020
 ---------------------

README.md

@@ -961,20 +961,116 @@ Additionally, if a search directory is specified in the form
   `--disable-wrapper-runpath`.
 
 * `--enable-dlopen`:
-  Build all of Open MPI's components as standalone Dynamic Shared
-  Objects (DSO's) that are loaded at run-time (this is the default).
-  The opposite of this option, `--disable-dlopen`, causes two things:
+  Enable loading of Open MPI components as standalone Dynamic
+  Shared Objects (DSOs) that are loaded at run-time.  This option is
+  enabled by default.
 
-  1. All of Open MPI's components will be built as part of Open MPI's
-     normal libraries (e.g., `libmpi`).
-  1. Open MPI will not attempt to open any DSO's at run-time.
+  The opposite of this option, --disable-dlopen, causes the following:
 
-  Note that this option does *not* imply that OMPI's libraries will be
-  built as static objects (e.g., `libmpi.a`).  It only specifies the
-  location of OMPI's components: standalone DSOs or folded into the
-  Open MPI libraries.  You can control whether Open MPI's libraries
-  are build as static or dynamic via `--enable|disable-static` and
-  `--enable|disable-shared`.
+  1. Open MPI will not attempt to open any DSOs at run-time.
+  1. configure behaves as if the --enable-mca-static argument was set.
+  1. configure will ignore the --enable-mca-dso argument.
+
+  See the description of --enable-mca-static / --enable-mca-dso for
+  more information.
+
+  Note that this option does *not* change how Open MPI's libraries
+  (libmpi, for example) will be built.  You can change whether Open
+  MPI builds static or dynamic libraries via the
+  --enable|disable-static and --enable|disable-shared arguments.
+
+* `--enable-mca-dso[=LIST]` and `--enable-mca-static[=LIST]`
+  These two options, along with --enable-mca-no-build, govern the
+  behavior of how Open MPI's frameworks and components are built.
+
+  The --enable-mca-dso option specifies which frameworks and/or
+  components are built as Dynamic Shared Objects (DSOs).
+  Specifically, DSOs are built as "plugins" outside of the core Open
+  MPI libraries, and are loaded by Open MPI at run time.
+
+  The --enable-mca-static option specifies which frameworks and/or
+  components are built as part of the core Open MPI libraries (i.e.,
+  they are not built as DSOs, and therefore do not need to be
+  separately discovered and opened at run time).
+
+  Both options can be used one of two ways:
+
+  1. --enable-mca-OPTION (with no value)
+  1. --enable-mca-OPTION=LIST
+
+  --enable-mca-OPTION=no or --disable-mca-OPTION are both legal
+  options, but have no impact on the selection logic described below.
+  Only affirmative options change the selection process.
+
+  LIST is a comma-delimited list of Open MPI frameworks and/or
+  framework+component tuples.  Examples:
+
+  * "btl" specifies the entire BTL framework
+  * "btl-tcp" specifies just the TCP component in the BTL framework
+  * "mtl,btl-tcp" specifies the entire MTL framework and the TCP
+     component in the BTL framework
+
+  Open MPI's configure script uses the values of these two options
+  when evaluating each component to determine how it should be built
+  by evaluating these conditions in order:
+
+  1. If an individual component's build behavior has been specified
+     via these two options, configure uses that behavior.
+  1. Otherwise, if the component is in a framework whose build
+     behavior has been specified via these two options, configure uses
+     that behavior.
+  1. Otherwise, configure uses the global default build behavior.
+
+  At each level of the selection process, if the component is
+  specified to be built as both a static and dso component, the static
+  option will win.
+
+  Note that as of Open MPI v5.0.0, configure's global default is to
+  build all components as static (i.e., part of the Open MPI core
+  libraries, not as DSO's).  Prior to Open MPI v5.0.0, the global
+  default behavior was to build most components as DSOs.
+
+  Also note that if the --disable-dlopen option is specified, then
+  Open MPI will not be able to search for DSOs at run time, and the
+  value of the --enable-mca-dso option will be silently ignored.
+
+  Some examples:
+
+  1. Default to building all components as static (i.e., as part of
+     the Open MPI core libraries -- no DSOs):
+
+     $ ./configure
+
+  1. Build all components as static, except the TCP BTL, which will be
+     built as a DSO:
+
+     $ ./configure --enable-mca-dso=btl-tcp
+
+  1. Build all components as static, except all BTL components, which
+     will be built as DSOs:
+
+     $ ./configure --enable-mca-dso=btl
+
+  1. Build all components as static, except all MTL components and the
+     TCP BTL component, which will be built as DSOs:
+
+     $ ./configure --enable-mca-dso=mtl,btl-tcp
+
+  1. Build all BTLs as static, except the TCP BTL, as the
+     <framework-component> option is more specific than the
+     <framework> option:
+
+     $ ./configure --enable-mca-dso=btl --enable-mca-static=btl-tcp
+
+  1. Build the TCP BTL as static, because the static option at the
+     same level always wins:
+
+     $ ./configure --enable-mca-dso=btl-tcp --enable-mca-static=btl-tcp
+
+* `--enable-mca-no-build=LIST`:
+  Comma-separated list of `<framework>-<component>` pairs that will not be
+  built. For example, `--enable-mca-no-build=btl-portals,oob-ud` will
+  disable building the portals BTL and the ud OOB component.
 
 * `--disable-show-load-errors-by-default`:
   Set the default value of the `mca_base_component_show_load_errors`
@@ -1017,11 +1113,6 @@ Additionally, if a search directory is specified in the form
   these libraries for their own purposes. This option is *not*
   intended for typical users of Open MPI.
 
-* `--enable-mca-no-build=LIST`:
-  Comma-separated list of `<type>-<component>` pairs that will not be
-  built. For example, `--enable-mca-no-build=btl-portals,oob-ud` will
-  disable building the portals BTL and the ud OOB component.
-
 
 ### Networking support / options
 

config/opal_mca.m4

@@ -12,7 +12,7 @@ dnl Copyright (c) 2004-2005 The Regents of the University of California.
 dnl                         All rights reserved.
 dnl Copyright (c) 2010-2016 Cisco Systems, Inc.  All rights reserved.
 dnl Copyright (c) 2013-2017 Intel, Inc. All rights reserved.
-dnl Copyright (c) 2018      Amazon.com, Inc. or its affiliates.
+dnl Copyright (c) 2018-2021 Amazon.com, Inc. or its affiliates.
 dnl                         All Rights reserved.
 dnl $COPYRIGHT$
 dnl
@@ -68,16 +68,16 @@ AC_DEFUN([OPAL_MCA],[
                         type-component pairs that will be built as
                         run-time loadable components (as opposed to
                         statically linked in), if supported on this
-                        platform.  The default is to build all components
-                        as DSOs.]))
+                        platform.]))
     AC_ARG_ENABLE(mca-static,
         AC_HELP_STRING([--enable-mca-static=LIST],
                        [Comma-separated list of types and/or
                         type-component pairs that will be built statically
                         linked into the library.  The default (if DSOs are
                         supported) is to build all components as DSOs.
                         Enabling a component as static disables it
-                        building as a DSO.]))
+                        building as a DSO.  The default is to build all
+                        components staticly.]))
     AC_ARG_ENABLE(mca-direct,
         AC_HELP_STRING([--enable-mca-direct=LIST],
                        [Comma-separated list of type-component pairs that
@@ -166,15 +166,21 @@ AC_DEFUN([OPAL_MCA],[
     # resolution (prefer static) is done in the big loop below
     #
     AC_MSG_CHECKING([which components should be run-time loadable])
-    if test "$enable_static" != "no" || test "$OPAL_ENABLE_DLOPEN_SUPPORT" = 0; then
+    if test "$enable_static" != "no"; then
         DSO_all=0
-        msg=none
-    elif test -z "$enable_mca_dso" || test "$enable_mca_dso" = "yes"; then
-        DSO_all=1
-        msg=all
+        msg="none (static libraries built)"
+    elif test "$OPAL_ENABLE_DLOPEN_SUPPORT" = 0; then
+        DSO_all=0
+        msg="none (dlopen disabled)"
+    elif test -z "$enable_mca_dso"; then
+        DSO_all=0
+        msg=default
     elif test "$enable_mca_dso" = "no"; then
         DSO_all=0
         msg=none
+    elif test "$enable_mca_dso" = "yes"; then
+        DSO_all=1
+        msg=all
     else
         DSO_all=0
         ifs_save="$IFS"
@@ -195,12 +201,15 @@ AC_DEFUN([OPAL_MCA],[
     fi
 
     AC_MSG_CHECKING([which components should be static])
-    if test "$enable_mca_static" = "yes"; then
-        STATIC_all=1
-        msg=all
-    elif test -z "$enable_mca_static" || test "$enable_mca_static" = "no"; then
+    if test -z "$enable_mca_static" ; then
+        STATIC_all=0
+        msg=default
+    elif test "$enable_mca_static" = "no"; then
         STATIC_all=0
         msg=none
+    elif test "$enable_mca_static" = "yes"; then
+        STATIC_all=1
+        msg=all
     else
         STATIC_all=0
         ifs_save="$IFS"
@@ -705,17 +714,23 @@ AC_DEFUN([MCA_COMPONENT_COMPILE_MODE],[
         [str="STATIC_COMPONENT=\$STATIC_$2_$3"
          eval $str])
 
-    # Setup for either shared or static
-    if test "$STATIC_FRAMEWORK" = "1" || \
-       test "$STATIC_COMPONENT" = "1" || \
-       test "$STATIC_all" = "1" ; then
-        $4="static"
-    elif test "$SHARED_FRAMEWORK" = "1" || \
-         test "$SHARED_COMPONENT" = "1" || \
-         test "$DSO_all" = "1"; then
-        $4="dso"
+    # Look for the most specific specifier between static/dso.  If
+    # there is a tie (either neither or both specified), prefer
+    # static.
+    if test "$STATIC_COMPONENT" = "1"; then
+        $4=static
+    elif test "SHARED_COMPONENT" = "1"; then
+        $4=dso
+    elif test "$STATIC_FRAMEWORK" = "1"; then
+        $4=static
+    elif test "$SHARED_FRAMEWORK" = "1"; then
+        $4=dso
+    elif test "$STATIC_all" = "1"; then
+        $4=static
+    elif test "$DSO_all" = "1"; then
+        $4=dso
     else
-        $4="static"
+        $4=static
     fi
 
     AC_MSG_CHECKING([for MCA component $2:$3 compile mode])