Skip to content

devel CreateComponent

Jump to bottom
Jeff Squyres edited this page Sep 2, 2014 · 4 revisions

Create a new MCA component

This wiki page goes through the basics of creating a new component in an existing Open MPI Modular Component Architecture (MCA) framework. It is assumed that the reader already has at least some familiarity with the internals of Open MPI.

NOTE: This page describes the most recent version of Open MPI's "autogen" process. For older versions, refer to the archived versions of this wiki page:

Related wiki pages

  • Note that OMPI's top-level "autogen.pl" is pivotal to the discovery, configuration, and building of components in the Open MPI source code tree. Be sure to see the autogen.pl wiki page for details about the role of autogen.pl.
  • How to create a new MCA framework.

Removing a component

As you'll see below, a component is solely contained within a single directory tree. As such, removing a component is accomplished via "rm -rf" of the component directory tree and re-running autogen.pl and configure.

Since it would be an abstraction violation for one component to refer to another (with the sole exception of "common" pseudo-components), it should be to "rm -rf" a component without changing any other part of the OMPI source code tree.

Creating the component

There are several different options when creating a new component; this document walks through the most common: a component that has some build-time requirements (i.e., it can't be built unless one or more support libraries are present, such as those for a specialized network type) that are checked via the top-level Open MPI configure script and some run-time requirements.

Note that every framework has different rules and interfaces that its components must adhere to. This document does not attempt to cover those rules; see the framework's main interface .h file for details.

  1. This document assumes that you are creating a component in the framework named "foo".
  2. Pick a component name. It must obey the following guidelines:
    1. Be valid as a C variable name (no spaces/punctuation other than "_", start with a letter, etc.)
    2. Be all lower case
    3. Be unique across all other components in this MCA framework (i.e., in the "foo" framework). It is permissible to have another component with the same name in a different framework (and is actually somewhat desirable, if all the components of the same name shared characteristics, such as support the same specialized network).
    4. Is at least somewhat descriptive (be friendly to your fellow developers!)
  3. Create a directory with the component name in /mca/foo/. For the purposes of this document, we'll assume that your component name is "bar" in the ompi project (i.e., ompi/mca/foo/bar/).
  4. Note that the "configure.params" file is no longer used (prior versions of the build system used it). If a "configure.params" file is present in the component directory, it will be ignored.

The entire source code of the component must go into the single directory tree ompi/mca/foo/bar/. The only exception to this rule is "common" pseudo-components, described below.

Configuring the component

A component has multiple choices about how to configure itself (which is different than the decision about how to build itself -- see below). Specifically, the top-level OMPI configure script must be informed as to whether a given component wants to build or not.

This decision is made by having at least one of the following files in the component's directory (each of which is described in detail below):

  1. configure.m4 '''(this is the preferred file to use)'''
  2. configure.in or configure.ac
  3. Makefile.am (simple)

If none of these files are present, or if the file ".ompi_ignore" is present in the component directory, autogen.pl will skip this directory and it will not be included in the build system.

"Simple" components can have only a Makefile.am with no configure.* file. This is effectively a shortcut to ensuring that your component will always be built (i.e., configure will never decide not to build your component).

The contents of the various "configure.*" files are described below.

Using a "configure.m4" file

''Using a configure.m4 file is the preferred method for configuring components that are in the mainline Open MPI code base.''

If the component has a configure.m4 file, it is slurped into the top-level OMPI configure script by autogen.pl. configure.m4 can contain a few different hooks that are executed at strategic points during configure that determine both how the component is configured and whether it will be built or not.

Note that in some frameworks, the concept of a "build priority" is used to determine which component is built. Components in these frameworks must use a configure.m4 file; they cannot use a configure.in, configure.ac, or implicit configure file.

Some components must also force themselves to always be compiled statically (e.g., slurped up into their higher-level lib.la files) or as a DSO. Components can override the default built settings, but only via the configure.m4 file.

  • It is required that the configure.m4 file define an m4 macro named MCA_<project>_<framework>_<component>_CONFIG (in this example, MCA_ompi_foo_bar_CONFIG). The macro can execute any of the normal Autoconf / Automake tests and determine if it can built itself. The macro is also passed two parameters: $1 and $2. $1 should be executed if the macro determines that the component wants to be built; $2 should be executed otherwise.
    • You will need to invoke AC_CONFIG_FILES to have the top-level configure generate any relevant files (e.g., Makefile).
    • It is permissible to invoke AC_ARG_WITH and AC_ARG_ENABLE, which will add --with/--without and --enable/--disable command line options to Open MPI's top-level configure script. These can be useful mechanisms to pass in parameters to your component's configure macro. For example, the openib BTL component invokes AC_ARG_WITH to create the --with-openib command line parameter. If a parameter is supplied, it is assumed to be the path location where the OpenFabrics header files and libraries are located. NOTE: If you use AC_ARG_WITH or AC_ARG_ENABLE, please also use AC_HELP_STRING to make a proper help string for when users invoke "configure --help".
    • You can generally call any AC_ or AM_ built-in tests that you want to determine if your component can build (e.g., test for the presence of libraries and/or header files, etc.). The only notable exception is AM_CONDITIONAL: '''you must not invoke AM_CONDITIONAL in the MCA_<project>_<framework>_<component>_CONFIG macro! ''' (you will get Automake errors if you do). Instead, you can invoke AM_CONDITIONAL in the MCA_<project>_<framework>_<component>_POST_CONFIG macro, described below.
    • You may AC_DEFINE[UNQUOTED] any values that you wish, but should generally prefix them with upper-case versions of "" (for example, "OMPI_FOO_BAR_WANT_GADGETS"), since preprocessor macros have only a single namespace.
    • Similarly, you may AC_SUBST any values that you wish, but must also obey the prefix rule. For example, it is permissible to AC_SUBST([ompi_foo_bar_CPPFLAGS]). You can then use this value in your component's Makefile.am (for example -- assuming that Open MPI's top-level configure is generating your Makefile). This can be useful if you need to pass additional -I, -L, and/or -l flags to build your component. See various other Open MPI component Makefile.am's for examples of this technique, such as the AC_SUBST's at the bottom of source: trunk/ompi/mca/btl/openib/configure.m4 and their corresponding use in source: trunk/ompi/mca/btl/openib/Makefile.am.
    • It is highly suggested to be verbose in the output of configure. Use AC_MSG_CHECKING and AC_MSG_RESULT to indicate what you are checking so that a human can read the output and generally surmise what is happening, and how/why individual tests succeed or fail.
    • If you set any of the compiler/assembler/linker flags in your configure.m4 script, you must reset them back to their original values before returning. For example (but not limited to): CPPFLAGS, CFLAGS, CXXFLAGS, FFLAGS, FCFLAGS, LDFLAGS, LIBS.
    • If your component cannot compile on a given platform, do not call AC_MSG_ERROR. AC_MSG_ERROR will abort the top-level OMPI configure script. Instead, just execute $2, which will cause OMPI to ignore this component and proceed with the rest of the build.
    • However, it is strongly suggested that if a specific option is requested (e.g., via a --with or --enable option), and that option is not possible, then you should AC_MSG_WARN to explain the problem and then AC_MSG_ERROR to abort. Specifically: if a user specifically asks for something and you cannot deliver it, it's an abortable error. But default configurations that will not build are not errors; it is perfectly acceptable to skip a component in a build that the user didn't specifically ask for.
    • OMPI defines several helper m4 macros that may be useful, for example:
      • OMPI_VAR_SCOPE_PUSH / OMPI_VAR_SCOPE_POP: For listing all the temporary shell variables that you will use in your macro. The _POP macro will unset all of them (to be social/safe with the rest of the Bourne shell code in configure).
      • OMPI_CHECK_PACKAGE: Checks for header files and corresponding library files (e.g., do all the work for checking for the OpenFabrics header and library files). See the comments in source: trunk/config/ompi_check_package.m4 for details.
  • The component can optionally define an MCA_<project>_<framework>_<component>_POST_CONFIG macro if it needs to invoke AM_CONDITIONAL. See below for an example.
    • The _POST_CONFIG macro must not create any output during execution unless there was an error.
    • The _POST_CONFIG macro should not assume that the _CONFIG macro was executed.
  • In frameworks that define a build priority, the configure.m4 must define MCA_<project>_<framework>_<component>_PRIORITY to be a single integer in the range of [0, 100]. For example:
# Set the component's priority to 40
AC_DEFUN([MCA_ompi_foo_bar_PRIORITY], [40])
  • Components that require being built statically or dynamically can override defaults by definining an MCA_<project>_<framework>_<component>_COMPILE_MODE macro. $2 will contain the framework name, $3 will contain the component name, and $4 should be assigned the value "static" or "dso". For example:
# Force this component to always be built statically
AC_DEFUN([MCA_ompi_foo_bar_COMPILE_MODE], [
    AC_MSG_CHECKING([for MCA component $2:$3 compile mode])
    $4="static"
    AC_MSG_RESULT([$$4])
])
  • For the most part, components are not allowed to influence the build system outside of that component. As mentioned, they should not set CPPFLAGS, CFLAGS, etc. Instead, they should set AC_SUBST'ed variables which are then used in the Makefile.am to set AM_CPPFLAGS (or similar). By convention, these variables are named <framework>_<compnent>_<flag>, although they may be named anything.
  • The MCA system automatically handles passing LDFLAGS and LIBS from a component to the top-level library (libmpi.so, for example) if the component is built statically into the top-level library. However, if the top-level library is also built statically, those flags need to be passed to the wrapper compiler. A component may add LDFLAGS and LIBS to the wrapper compiler flags by setting <framework>_<component>_WRAPPER_EXTRA_(LDFLAGS, LIBS) in the CONFIG macro. If neither of these are set, the system will try copying <framework>_<component>_(LDFLAGS, LIBS) instead. For the most part, a component should not have to influence the CPPFLAGS of the wrapper compilers. However, static frameworks such as hwloc typically require some extra includes. If a framework is STOP_AT_FIRST, the component is static, and devel headers are enabled, <framework>_<component>_WRAPPER_EXTRA_CPPFLAGS will be copied into the wrapper CPPFLAGS.

Here's a simple configure.m4 example with lots of comments:

# This macro is executed to determine if the component wants to be
# included in the Open MPI build process.  It is only executed if the
# user did not exclude this component and/or framework via the
# --with-mca-no-build OMPI configure command line switch (this is why
# you can't put AM_CONDITIONALs in this macro: the whole macro is
# executed conditionally).

AC_DEFUN([MCA_ompi_foo_bar_CONFIG],[
     OMPI_VAR_SCOPE_PUSH([ompi_foo_bar_message])
     # We need this directory's Makefile.am to generate Makefile
     AC_CONFIG_FILES([ompi/mca/foo/bar/Makefile])

     # We need the header file "gadgets.h" to build this component
     AC_CHECK_HEADERS([gadgets.h], 
                      [# If we have gadgets.h, see if we have the
                       # optional gadgets/froobles.h file
                       ompi_foo_bar_message=happy
                       AC_CHECK_HEADERS([gadgets/froobles.h],
                                        [ompi_foo_bar_have_froobles=1
                                         ompi_foo_bar_message="super happy"])
                       ompi_foo_bar_happy=1],
                      [ompi_foo_bar_message=unhappy
                       ompi_foo_bar_happy=0])

     # This is a contrived message/example just to use
     # $ompi_foo_bar_message, just so that we can
     # OMPI_VAR_SCOPE_PUSH/POP.
     AC_MSG_CHECKING([for bar happiness])
     AC_MSG_RESULT([$ompi_foo_bar_message])

     # Execute $1 if everything was happy above; $2 otherwise.  You
     # can imagine other logic flows to execute $1 or $2; this is just
     # one example.
     AS_IF([test "$ompi_foo_bar_happy" = "1"], [$1], [$2])
     AC_VAR_SCOPE_POP
])

# This macro is *always* executed during configure, and it is
# therefore safe to execute AM_CONDITIONAL in it.  $1 will be "1" if
# the MCA_ompi_foo_bar_CONFIG macro was executed.  Hence, you likely
# want to ensure to test the value of $1 in the AM_CONDITIONAL test
# clause (good defensive programming).

AC_DEFUN([MCA_ompi_foo_bar_POST_CONFIG], [
     # If we need to change build behavior depending on whether we
     # have gadgets/froobles.h, we can AM_CONDITIONAL with the result
     # of the test from the main _CONFIG macro, above.
     AM_CONDITIONAL([MCA_ompi_foo_bar_HAVE_FROOBLES], 
                    [test $1 -eq 1 -a "x$ompi_foo_bar_have_froobles" = "x1"])
])

Using a "configure.in" or "configure.ac" file

If the component has its own configure.in or configure.ac file, OMPI's autogen.pl will turn it into a full configure executable that is invoked during the OMPI build process.

  • If the componet has an autogen.sh or autogen.pl executable, then Open MPI's autogen.pl will invoke it. The autogen.pl|sh executable must return an exit status of 0 if it succeeds.
  • If no autogen.pl|sh executable is present in the top-level directory of the component, Open MPI's autogen.pl will invoke "autoreconf -ivf" to generate the component's configure script.

If no executable named "configure" is emitted as a result of this step, autogen.pl will abort.

At build time, Open MPI will invoke the resulting configure script. The exit status of the configure executable determines what happens to that component:

  • If the component configure executable returns 0, the top-level OMPI configure script assumes that the component wants to be built, and therefore the relevant "make" targets will traverse into the component directory during the build process.
  • Any other return status results in the top-level OMPI configure script ignoring that component during the build process.

Using no configure.* file at all

If your component meets the following conditions, it does not require any configure.* file:

  • It can be built in every environment
  • The top-level component directory contains a single Makefile.am that can build the entire component

To be clear, if autogen.pl sees a component directory with a Makefile.am but no configure.* file, it will simply invoke AC_CONFIG_FILES([///Makefile]) and assume that your component can be built in every environment.

Source code of the component

Generally, the source code is entirely up to you. :-)

However, the source code that is specific to this component must be entirely within the component directory (with the one exception of "common" pseudo-components, described below). That being said, you can call any library function in your component's library layer and below. For example, if your component is in the OMPI layer, it can call any OMPI, ORTE, or OPAL library call. If your component is in the ORTE layer, then it can only call any ORTE or OPAL library call (not OMPI).

It is illegal for components to directly call functions in other components (except for common pseudo-components; see below). Do not link your component against any other component (except for common pseudo-components) or assume that you can call any other component's internal functions. You must use published framework interfaces to call functions in other components.

You can have whatever directory layout you want in your component; many OMPI components are a single, top-level directory, but some have a multi-level directory structure. The specific directory layout of your component is irrelevant; the main requirement is that the output library/DSO must be in the top-level directory (see "Building the component", below).

Building the component

If a component was configured successfully, it can build itself however it wants. The only requirement is that it supports standard Automake "make" targets, such as (but not limited to):

  • all
  • install
  • clean
  • distclean
  • dist
  • test

Specifically, OMPI's top-level build process will traverse into the component's directory with any of the standard Automake "make" targets. These targets are expected the work in the same way as Automake-generated Makefiles.

Most (all?) components in the Open MPI tree have Makefile.am's and add themselves to the list of files to be generated by OMPI's top-level configure script (via AC_CONFIG_FILES in the MCA__CONFIG m4 macro).

Components are built in either "DSO" mode or static mode (never both simultaneously):

  • If the component is building in dynamic shared object (DSO) mode, meaning that it creates a standalone dynamic library file that is dynamically opened at run-time, it must create a DSO named "mca__.", where is whatever extension is relevant for that platform (e.g., "so", "dylib", "dll", etc.).
  • If the component is building in static mode, meaning that it will be included in the upper-level project library, it must create a Libtool convenience library named "libmca__.la".

NOTE: It is possible that these output filenames may change in the future in order to simplify some of the logic in component Makefile.am's.

Currently, the only way to tell which way a component is to be built is via the AM_CONDITIONAL OMPI_BUILD_<project>_<framework>_<component>_DSO. If it is true, the component is to be built as a DSO. Otherwise, it needs to be built statically. '''This may be problematic for components that use their own configure scripts. This may need to be fixed, if anyone cares! ''' (it's not a huge problem for the mainline OMPI code base because all components use the configure.m4 method)

There are many good examples of component Makefile.am's in the OMPI source tree. Two such examples are:

  • source: trunk/ompi/mca/btl/tcp/Makefile.am: A "plain vanilla" Makefile.am that simply lists several header and source files and builds them into either a DSO or a Libtool convenience library.
  • source: trunk/ompi/mca/btl/openib/Makefile.am: A fairly complex Makefile.am, that has all the same elements as the TCP BTL makefile, but also includes features such as:
    • Flex-generated source code
    • AMCA parameter file installation
    • Run-time help message file installation
    • Header and source files for the component itself
    • Conditional compilation of several component source files
    • Dependency on external libraries, and therefore adds CPPFLAGS, LDFLAGS, and LIBS to the component build process

"Common" pseudo-components

It is not uncommon to need to share some common code between multiple different components in different frameworks. For example, OpenFabrics-based components in both the point-to-point and collective MPI frameworks may want to share queue pair management code. Rather than duplicating this code in each component, it may be better to put this code in a single OpenFabrics "common" component.

NOTE: There is currently a limitation that the "common" pseudo-framework can only exist in the Open MPI project (i.e., "ompi/"). The ORTE and OPAL trees do not have a "common" pseudo-component. If you need a "common" pseudo-component in ORTE or OPAL, please send mail to the OMPI developer's mailing list.

The common framework in the OMPI project (i.e., ompi/mca/common) is unlike all other frameworks. There are no rules about interfaces, there are no requirements for framework open and close functions, there is no ompi/mca/common/base directory, etc. Simply put, the "common" tree is a repository for code shared between components in any OMPI project framework.

That being said, pseudo-components in the common framework do abide by the same configure.m4 / configure.ac|in / etc. rules described above. As such, for code to be compiled in the common framework, each pseudo-component must configure successfully (just like a real component).

Also just like real components, pseudo-components can be built as DSOs or statically. When built statically, the code is slurped into the upper-level libmpi.la and is therefore available to any component that wishes to use that code. When built as a DSO, you probably want to link relevant components against the desired common pseudo-component DSO.

A lengthy comment in source: trunk/ompi/mca/common/sm/Makefile.am explains the rationale and how it works. This Makefile.am also provides a template for how to compile common pseudo-components.

For example, say that the "yow" component in the "foo" framework and the "yow" component in the "bar" framework (both in the "ompi" project) need to share some code.

  1. A developer can create the ompi/mca/common/yow/ pseudo-component and put some code in there that is used by both foo::yow and bar::yow components.
  2. Both the foo::yow and bar::yow components can link against the yow pseudo-component by adding $(top_ompi_builddir)/ompi/mca/common/yow/libmca_common_yow.la to their LIBADD line in the DSO case. See source: trunk/ompi/mca/btl/sm/Makefile.am as an example.
Clone this wiki locally