README

GEOPM - Global Extensible Open Power Manager
============================================

DISCLAIMER
----------
SEE COPYING FILE FOR LICENSE INFORMATION.

LAST UPDATE
-----------
2018 February 28

Christopher Cantalupo <christopher.m.cantalupo@intel.com> <br>

WEB PAGES
---------
http://geopm.github.io <br>
http://geopm.github.io/man/geopm.7.html

SUMMARY
-------
Global Extensible Open Power Manager (GEOPM) is an extensible power
management framework targeting high performance computing.  The library can be
extended to support new control algorithms and new hardware power management
features.  The GEOPM package provides built in features ranging from static
management of power policy for each individual compute node, to dynamic
coordination of power policy and performance across all of the compute nodes
hosting one MPI job on a portion of a distributed computing system.  The
dynamic coordination is implemented as a hierarchical control system for
scalable communication and decentralized control.  The hierarchical control
system can optimize for various objective functions including maximizing
global application performance within a power bound.  The root of the control
hierarchy tree can communicate through shared memory with the system resource
management daemon to extend the hierarchy above the individual MPI job level
and enable management of system power resources for multiple MPI jobs and
multiple users by the system resource manager.  The geopm package provides the
libgeopm library, the libgeopmpolicy library, the geopmctl application and the
geopmpolicy application.  The libgeopm library can be called within MPI
applications to enable application feedback for informing the control
decisions.  If modification of the target application is not desired then the
geopmctl application can be run concurrently with the target application.  In
this case, target application feedback is inferred by querying the hardware
through Model Specific Registers (MSRs).  With either method (libgeopm or
geopmctl), the control hierarchy tree writes processor power policy through
MSRs to enact policy decisions.  The libgeopmpolicy library is used by a
resource manager to set energy policy control parameters for MPI jobs.  Some
features of libgeopmpolicy are available through the geopmpolicy application
including support for static control.

TRAVIS CI
---------
[![Build Status](https://travis-ci.org/geopm/geopm.svg)](https://travis-ci.org/geopm/geopm)

The geopm public github project has been integrated with Travis
continuous integration.

http://travis-ci.org/geopm/geopm

All pull requests will be built and tested automatically by Travis.
Additionally, pushes to the public dev and master branches which pass
testing in Travis will trigger build and test in the OpenSUSE Build
System (OBS) for a variety of Linux distributions.  These provide yum
and zypper repositories which can be used to install the geopm
software.

INSTALL
-------
The products of OBS triggered by Travis CI can be installed directly
on a number of Linux distributions through yum or zypper.  Build and
test are currently passing on CentOS 7, RHEL 7, SLES 12, SLES 12-1,
Fedora 21, Fedora 22 and Fedora 23.  Release 0.2.0 is the first to
include distribution on OBS.  Pushes to the github public master
branch are done only after a release has been tagged, and these tagged
releases are published in yum and zypper repositories at the link
below:

http://download.opensuse.org/repositories/home:/cmcantalupo:/geopm

The tagged releases should be more stable and more accurately
documented than the tip of the public dev branch which is updated
frequently.  These frequent pushes to the public development branch
can also be used as yum and zypper repositories.  These are published
at the link below:

http://download.opensuse.org/repositories/home:/cmcantalupo:/geopm-dev

Use of these repositories should be helpful for developers of geopm
who are willing to update the installed version frequently.

### Red Hat
To install the yum repository follow the commands below.  Select the
DISTRO appropriate for you system.  Select the BRANCH as "geopm" or
"geopm-dev" to follow either the "master" or "dev" branches of
repository hosted at <http://github.com/geopm/geopm>.

    DISTRO=# Select from: "CentOS_7" "RHEL_7" "Fedora_23" "Fedora_24"
    BRANCH=# Select from: "geopm" or "geopm-dev"
    ROOT="home:/cmcantalupo:"
    REPO=$(echo $ROOT | sed 's|/||')
    wget http://download.opensuse.org/repositories/${ROOT}/${BRANCH}/${DISTRO}/${REPO}${BRANCH}.repo
    sudo install --mode 644 ${REPO}${BRANCH}.repo /etc/yum.repos.d/${BRANCH}.repo
    unset DISTRO BRANCH ROOT REPO

After the yum repository has been added, you can simply install geopm
with yum:

    sudo yum install geopm # For headers install geopm-devel

To bring the system up to date with the tip of the public master or
dev branch on github (depending on which "BRANCH" was selected when
the yum repository was added:

    sudo yum update geopm # For headers update geopm-devel

### SUSE
To install the zypper repository follow the commands below.  Select the
DISTRO appropriate for you system.  Select the BRANCH as "geopm" or
"geopm-dev" to follow either the "master" or "dev" branches of
repository hosted at <http://github.com/geopm/geopm>.

    DISTRO=# Select from: "SLE_12" or "SLE_12_SP1"
    BRANCH=# Select from: "geopm" or "geopm-dev"
    ROOT="home:/cmcantalupo:"
    REPO=$(echo $ROOT | sed 's|/||')
    sudo zypper addrepo http://download.opensuse.org/repositories/${ROOT}/${BRANCH}/${DISTRO}/${REPO}${BRANCH}.repo ${BRANCH}
    sudo zypper refresh
    unset DISTRO BRANCH ROOT REPO

After the zypper repository has been added, you can simply install
geopm with zypper:

    sudo zypper install geopm # For headers install geopm-devel

To bring the system up to date with the tip of the public master or
dev branch on github (depending on which "BRANCH" was selected when
the zypper repository was added:

    sudo zypper update geopm # For headers update geopm-devel

PYTHON INSTALL
--------------
The GEOPM python tools are packaged in the RPMs described above, but
they are also available from PYPI as the `geopmpy` package.  For
example to install the geopmpy package into your home directory run
the following command:

    pip install --user geopmpy

Note this installs only the GEOPM python tools and does not install
the full GEOPM runtime.

BUILD REQUIREMENTS
------------------
In order to build the geopm package from source, the below
requirements must be met.

The geopm package requires a compiler that supports the MPI 2.2 and
C++11 standards.  These requirements can be met by using GCC version
4.7 or greater and installing the openmpi-devel package version 1.7 or
greater on RHEL or SLES Linux.  Additionally building the geopm
package requires the hwloc library. Documentation creation including
manpages further requires the rubygems package, and the ruby-devel
package.

RHEL:

    yum install openmpi-devel hwloc-devel ruby-devel rubygems

SUSE:

    zypper install openmpi-devel hwloc-devel ruby-devel rubygems

Alternatively these can be installed from source, and an alternate MPI
implementation to OpenMPI can be selected.  See

    ./configure --help

for details on how to use non-standard install locations for build
requirements through the

    ./configure --with-<feature>

options.


BUILD INSTRUCTIONS
------------------
The source code can be rebuilt from the source RPMs available from the
links in the INSTALL section using rpmbuild.  To build from the git
repository follow the instructions below.

To build all targets and install it in a "build/geopm" subdirectory of your
home directory run the following commands:

    ./autogen.sh
    ./configure --prefix=$HOME/build/geopm
    make
    make install

An RPM can be created on a RHEL or SUSE system with the

    make rpm

target.  Note that the --with-mpi-bin option may be required to inform
configure about the location of the MPI compiler wrappers.  The following
command may be sufficient to determine the location:

    dirname $(find /usr -name mpicc)

To build in an environment without support for OpenMP (i.e. clang on Mac OS X)
use the

    ./configure --disable-openmp

option.  The

    ./configure --disable-mpi

option can be used to build only targets which do not require MPI.  By default
MPI targets are built.


RUN REQUIREMENTS
----------------
We are targeting SLES12 and RHEL7 distributions for functional runtime
support.  There are several runtime requirements that can be obtained
from those distributions including the OpenMPI implementation of MPI
and the hwloc library.  To install follow the instructions below for
your Linux distribution.

RHEL:

    yum install openmpi hwloc

SUSE:

    zypper install openmpi hwloc

### SYSTEMD CONFIGURATION
In order for GEOPM to properly use shared memory to communicate between the
Controller and the application, it may be necessary to alter the configuration
for systemd.  The default behavior of systemd is to clean-up all interprocess
communication for non-system users.  This causes issues with GEOPM's
initialization routines for shared memory.  This can be disabled by
ensuring that `RemoveIPC=no` is set in `/etc/systemd/logind.conf`.  Most
Linux distributions change the default setting to disable this behavior.  More
information can be found [here](https://superuser.com/a/1179962).

### MSR DRIVER
The msr-safe kernel driver must be loaded at runtime to
support user level read and write of white-listed MSRs.  The source
code for the driver can be found here:
<https://github.com/scalability-llnl/msr-safe>.

Alternately you can run geopm as root with the standard msr driver loaded:

    modprobe msr

Note that other Linux mechanisms for power management can interfere
with geopm, and these must be disabled.  We suggest the following:

    echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

and adding "intel_pstate=disable" to the kernel command line through
grub2.

### GEOPM MPI LAUNCH WRAPPER
The GEOPM package installs two commands: "geopmsrun" and "geopmaprun"
which are wrappers for the MPI launch commands "srun" and "aprun"
respectively.  These wrappers enable exactly the same command line
interface as the underlying launch command, but the wrapper extends
the interface with GEOPM specific options.  When the "--geopm-ctl"
option is passed to the wrapper it will launch the GEOPM runtime with
your application while enforcing the GEOPM requirements by
manipulating the options passed to the underlying launch command
("srun" or "aprun").  This includes handling the CPU affinity
requirements and getting the GEOPM control process on each node up and
ready to connect to the main compute application.  The wrapper is
documented in the geopmsrun(1) and geopmaprun(1) man pages.

If your system uses an MPI launch command other than "srun" or
"aprun", or if your application requires some specialized use of these
commands, then check the geopm_aprun(3) / geopm_srun(3) man page
description of the "--geopm-rm" option which selects from other
available launch mechanisms.  If the launch mechanism for your system
is not implemented, then affinity requirements must be enforced by the
user.  Also, please consult the geopm(7) man page for documentation of
the environment variables used by the GEOPM runtime that are otherwise
controlled by the wrapper script.

### CPU AFFINITY REQUIREMENTS
The GEOPM runtime requires that each MPI process of the application
under control is affinitized to distinct CPUs.  This is a strict
requirement for the runtime and must be enforced by the MPI launch
command.

Affinitizing the GEOPM control thread to a CPU that is distinct from
the application CPUs may improve performance of the application, but
this is not a requirement.  On systems where an application achieves
highest performance when leaving a CPU unused by the application so
that this CPU can be dedicated to the operating system, it is usually
best to affinitize the GEOPM control thread to this CPU designated for
system threads.

There are many ways to launch an MPI application, and there is no
single uniform way of enforcing MPI rank CPU affinities across
different job launch mechanisms.  Additionally, OpenMP runtimes, which
are associated with the compiler choice, have different mechanisms for
affinitizing OpenMP threads within CPUs available to each MPI process.
To complicate things further the GEOPM control thread can be launched
as an application thread or a process that may be part of the primary
MPI application or a completely separate MPI application.  For these
reasons it is difficult to document how to correctly affinitize
processes in all configurations.  Please refer to your site
documentation about CPU affinity for the best solution on the system
you are using.

## MPI LIBRARY PATH
On some systems it has been observed that if the path to libmpi.so is
not in the LD_LIBRARY_PATH or in the default load library path
(e.g. /usr/lib64), then the loading of plugins can fail on the
dlopen(3) call.  In the future we will be changing the way that
plugins are loaded to leverage library init functions instead of using
dlopen(3)/dlsym(3).  To work-around this problem, simply add the path
to libmpi.so to the LD_LIBRARY_PATH environment variable at runtime.

TESTING
-------
From within the source code directory, tests can be executed with the
"make check" target.  Some tests must be run as MPI applications, and
in order support these tests you must define the "MPIEXEC" environment
variable to whatever is appropriate for your your MPI runtime (default
is mpiexec).  The tests assume that this application will respect the
'-n' flag to set the number of MPI processes.  Note that '-n' is defined
by the MPI standard.  Each MPI dependent test is run with 16 MPI
processes.

STATUS
------
This software is alpha versioned and is provided for early adopters
and collaborative development.  We are very interested in feedback
from the community.  The runtime implements the infrastructure and
basic features targeted for GEOPM version 1.0.  We do not anticipate
changes to application facing interfaces of our libraries or the
command line interfaces of our executables until after version 1.0.
Before the beta release we will change the way signals and controls
are interfaced to decider plugins.  These changes will enable easier
and more efficient extension of signals and controls as well as
support for multiple domain definitions where signals and controls
pertain.  We will also be expanding our integration test feature
coverage, and our unit test line coverage before beta release.  See
TODO file for links to descriptions of forthcoming work.

Test coverage is incomplete.  The line coverage results from gcov as
reported by gcovr for the latest release can be found here:
<http://geopm.github.io/coverage/index.html>

ACKNOWLEDGMENTS
---------------
Development of the GEOPM software package has been partially funded
through contract B609815 with Argonne National Laboratory.