From 8f1c4c63f5ed9f09c31888d3ade07bc3e4949cb7 Mon Sep 17 00:00:00 2001 From: Kinshuk Date: Tue, 5 Nov 2024 17:15:22 -0700 Subject: [PATCH 1/7] update Ipopt documentation partially. Next step HSL. --- docs/Documentation/Applications/ipopt.md | 167 ++++++++++++++++------- 1 file changed, 121 insertions(+), 46 deletions(-) diff --git a/docs/Documentation/Applications/ipopt.md b/docs/Documentation/Applications/ipopt.md index a9f9d0fd9..6229f441a 100644 --- a/docs/Documentation/Applications/ipopt.md +++ b/docs/Documentation/Applications/ipopt.md @@ -14,75 +14,150 @@ grand_parent: Development IPOPT is commonly used in solving power flow, e.g., AC Optimal Power Flow, and controls problems. Please refer to their [project website](https://github.com/coin-or/Ipopt) for the source code. The documentation can be found [here](https://coin-or.github.io/Ipopt/index.html). -!!! info - IPOPT with HSL linear solvers is available as a module on Kestrel. Please see [IDAES Solvers](./idaes_solvers.md) for additional details. +!!! note + IPOPT with HSL linear solvers is available as a module on Kestrel. Please see [IDAES Solvers](./idaes_solvers.md) for additional details. We recommend using the system module for ease-of-use and only build if they do not meet your needs. ## Installation from source -!!! warning - The following installation instructions are for NREL's older cluster Eagle. We recommend users use the [IPOPT modules available on Kestrel](./idaes_solvers.md). Users will have to tweak the instructions below to build IPOPT from source on Kestrel. +!!! info + We advise build all applications on the compute node on an interactive session. Please see [Running Interactive Jobs](../Slurm/interactive_jobs.md#running-interactive-jobs) for additonal details. + +### Optional Pre-requisites -The default installation instructions can be found in the [IPOPT documentation here](https://coin-or.github.io/Ipopt/INSTALL.html). The remainder of the page describes what has worked for NREL HPC users. +We will build IPOPT using all prerequisites mentioned below. Users may pick and +choose depending on their needs. -We will use COIN-OR's [coinbrew](https://github.com/coin-or/coinbrew) repo to build IPOPT along with the dependencies ASL, HSL and Mumps libraries. +#### Metis +It is highly recommended to install [Metis](https://github.com/KarypisLab/METIS.git) +- Serial Graph Partitioning and Fill-reducing Matrix Ordering software to +improve the performance of linear solvers such as MUMPS and HSL. + +!!! warning + Using HSL linear solvers requires installing Metis. Metis is optional MUMPS. + +We will install Metis using Anaconda, however, it can also be installed from source. +To install using Anaconda, we will create a clean environment with nothing but Metis. +The conda environment is being constructed within a directory in `hpcapps` project on +Kestrel. + +```bash +module load conda +conda create -p /projects/hpcapps/kpanda/conda-envs/metis python +conda activate /projects/hpcapps/kpanda/conda-envs/metis +conda install conda-forge::metis +``` + +#### Coinbrew +[Coinbrew](https://github.com/coin-or/coinbrew) is a package manager to install +COIN-OR tools. It makes installing IPOPT and its dependencies easier. However, it +is not necessary to the installation if one clones the repositories individually. +A user can download `coinbrew` by running the following command + +```bash +wget https://raw.githubusercontent.com/coin-or/coinbrew/master/coinbrew +``` + +#### Intel oneAPI MKL +Intel oneAPI MKL provides BLAS and LAPACK libraries for efficient linear algebra. +Additionally, it also provides access to oneMKL PARDISO linear solver that is +compatible with Ipopt. !!! note - Follow the [instructions to setup the environment](../Development/Libraries/hsl.md#hpc) for HSL before proceeding with the steps below. - -1. `module load gcc/8.4.0 mkl` -2. Clone (or download) the [coinbrew](https://github.com/coin-or/coinbrew) repo. If you download the repo you may have to change the permissions on the `coinbrew` *script* before using it: `chmod u+x coinbrew/coinbrew` -3. `cd` into the directory -4. `./coinbrew fetch Ipopt@stable/3.13` - * This fetches the branch `stable/3.13` of the IPOPT repository as well as the dependencies COIN-OR repositories `ThirdParty-ASL`, `ThirdParty-HSL` and `ThirdParty-Mumps` (other versions of IPOPT can also be downloaded in this manner) -5. `cd ThirdParty/HSL` -6. Copy the HSL source code to the current directory and unpack it -7. Create a link called `coinhsl` that points to the HSL source code (or rename the directory) -8. Go back to coinbrew root directory: `cd ../..` -9. Configure and build everything: - - ```bash - ./coinbrew build Ipopt --disable-java --prefix="${MYAPPS}" --with-metis-cflags="-I${MYINC}" --with-metis-lflags="-L${MYLIB} -lmetis" --with-lapack-lflags="-L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_rt -lpthread -lm -ldl" --with-lapack-cflags="-m64 -I${MKLROOT}/include" ADD_CFLAGS="-march=skylake-avx512" ADD_FCFLAGS="-march=skylake-avx512" ADD_FFLAGS="-march=skylake-avx512" - ``` + oneMKL PARDISO is not available on Kestrel GPU nodes since they consist of AMD processors. - * `build Ipopt` tells `coinbrew` to configure and build IPOPT and its dependencies - * `--disable-java` says to build IPOPT without the java interface - * `--prefix` says to install the library in "${MYAPPS}" - * `--with-metis-cflags` gives the compiler the location of the metis header "metis.h" - * `--with-metis-lflags` gives the linker the location and name of the metis library - * `--with-lapack-lflags` gives the location of LAPACK and BLAS libraries as well as the needed linker lines. Here we are using Intel's single dynamic library interface (google "mkl single dynamic library" for more details on this). - * `ADD_CFLAGS`, `ADD_FCFLAGS` and `ADD_FFLAGS` say to use those extra flags when compiling C and fortran code, respectively. +### Installation -!!! tip - When linking with MKL libraries, Intel's [link line advisor](https://software.intel.com/content/www/us/en/develop/articles/intel-mkl-link-line-advisor.html) is extremely helpful. +In this demonstration, we will install Ipopt within +`/projects/msoc/kpanda/apps/Ipopt/install`. However, one is free to set their +install directory as they wish. Starting with the base working directory +`/projects/msoc/kpanda/apps/` we will do the following -!!! note - When compiling Julia with MKL libraries, the single dynamic library interface is used to link against. This is why we are also using that linking method. Using a different linking method will cause unusual behaviors when using IPOPT with Julia (e.g. through JuMP). +```bash +cd /projects/msoc/kpanda/apps/ # go into the base working directory +wget https://raw.githubusercontent.com/coin-or/coinbrew/master/coinbrew # install coinbrew +coinbrew fetch Ipopt # Fetch Ipopt and its dependencies +``` -## Usage +This will download 2 additional directories `Ipopt` and `ThirdParty`. +`ThirdParty`, furthermore, contains 3 subdirectories `ASL`, `HSL`, and `Mumps`. +The source code of all but `HSL` will be downloaded. -### Using Custom IPOPT with JuMP +Next, we will create our install directories and subdirectories + +```bash +mkdir -p /projects/msoc/kpanda/apps/Ipopt/install # create the install directory +cd /projects/msoc/kpanda/apps/Ipopt/install # enter the directory +mkdir bin lib include # create some subdirectories +``` + +We then add symbolic links to Metis in the install directory. !!! note - When running your custom IPOPT build on Kestrel, you will need to do two things: + If `libmetis.so` is in your `LD_LIBRARY_PATH` you do not need to do this step. + +```bash +cd /projects/msoc/kpanda/apps/Ipopt/install/lib +ln -s /projects/hpcapps/kpanda/conda-envs/metis/lib/libmetis.so libmetis.so +cd ../include +ln -s /projects/hpcapps/kpanda/conda-envs/metis/include/metis.h metis.h +cd /projects/msoc/kpanda/apps/ # go back base directory +``` + +Next, we will load additional modules. If users require oneMKL PARDISO or would +like to leverage intel performance optimization, run the following commands + +```bash +module load intel-oneapi-mkl +``` + +Alternatively, users can load the open source Netlib LAPACK using the command + +```bash +module load netlib-lapack # Please ensure you do not have intel-oneapi-mkl loaded +``` + +We will now copy the HSL source code tarball into +`/projects/msoc/kpanda/apps/ThirdParty/HSL/`, unpack it, and rename or (create a +symbolic link to the unpacked directory) as `coinhsl`. + +We are now ready to install Ipopt and its dependencies. Going back to the base +directory, we will run the following commands + +```bash +cd /projects/msoc/kpanda/apps/ # go back base directory +./coinbrew build Ipopt --disable-java \ +--prefix=/kfs2/projects/msoc/kpanda/apps/Ipopt/install \ +--with-metis \ +--with-metis-cflags=-I/projects/hpcapps/kpanda/conda-envs/metis/include \ +--with-metis-lflags="-L/projects/hpcapps/kpanda/conda-envs/metis/lib -lmetis" \ +--parallel-jobs 4 \ +--verbosity 4 \ +--reconfigure +``` + +## Usage - 1. Load the same MKL module you compiled against: - ```bash - module load mkl - ``` - 2. Add the directory containing IPOPT and HSL libraries to your LD_LIBRARY_PATH - ```bash - export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MYLIB} - ``` +The installed Ipopt is now ready to be used. We need to update our `PATH` AND +`LD_LIBRARY_PATH` environment variables. In our demonstrations this will be +```bash +export PATH=/projects/msoc/kpanda/apps/Ipopt/install/bin:${PATH} +export LD_LIBRARY_PATH=/projects/msoc/kpanda/apps/Ipopt/install/lib:${LD_LIBRARY_PATH} +``` + +!!! note + Do not forget to load `intel-oneapi-mkl` or `netlib-lapack` before running Ipopt else your runs will fail. + +### Using Custom IPOPT with JuMP To use our custom installation of IPOPT with `Ipopt.jl`, we do the following: 1. Open the Julia REPL and activate an environment that has IPOPT installed 2. Tell Julia and `Ipopt.jl` the location of our IPOPT library and executable ```julia - ENV["JULIA_IPOPT_LIBRARY_PATH"] = ENV["MYLIB"] - ENV["JULIA_IPOPT_EXECUTABLE_PATH"] = ENV["MYBIN"] + ENV["JULIA_IPOPT_LIBRARY_PATH"] = ENV["/projects/msoc/kpanda/apps/Ipopt/install/lib"] + ENV["JULIA_IPOPT_EXECUTABLE_PATH"] = ENV["/projects/msoc/kpanda/apps/Ipopt/install/bin"] ``` 3. Rebuild `Ipopt.jl` with the above environment variables set to pick up the new library and executable ```julia From 7f3a1019ce5d699450215df859b099acf34b4d4a Mon Sep 17 00:00:00 2001 From: Kinshuk Date: Wed, 6 Nov 2024 15:42:00 -0700 Subject: [PATCH 2/7] updated IPOPT and HSL documentation. --- docs/Documentation/Applications/ipopt.md | 65 ++++++-- .../Development/Libraries/hsl.md | 139 ++++++++---------- 2 files changed, 113 insertions(+), 91 deletions(-) diff --git a/docs/Documentation/Applications/ipopt.md b/docs/Documentation/Applications/ipopt.md index 6229f441a..42c5a1dc1 100644 --- a/docs/Documentation/Applications/ipopt.md +++ b/docs/Documentation/Applications/ipopt.md @@ -2,7 +2,7 @@ title: IPOPT postdate: October 27, 2020 layout: default -author: Jonathan Maack +author: Jonathan Maack, Kinshuk Panda description: How to install and use IPOPT with different programming languages parent: Libraries grand_parent: Development @@ -21,7 +21,7 @@ IPOPT is commonly used in solving power flow, e.g., AC Optimal Power Flow, and c ## Installation from source !!! info - We advise build all applications on the compute node on an interactive session. Please see [Running Interactive Jobs](../Slurm/interactive_jobs.md#running-interactive-jobs) for additonal details. + We advise build all applications on the compute node on an interactive session. Please see [Running Interactive Jobs](../Slurm/interactive_jobs.md#running-interactive-jobs) for additional details. ### Optional Pre-requisites @@ -29,17 +29,18 @@ We will build IPOPT using all prerequisites mentioned below. Users may pick and choose depending on their needs. #### Metis -It is highly recommended to install [Metis](https://github.com/KarypisLab/METIS.git) -- Serial Graph Partitioning and Fill-reducing Matrix Ordering software to + +It is highly recommended installing [Metis](https://github.com/KarypisLab/METIS.git) +- Serial Graph Partitioning and Fill-reducing Matrix Ordering software to improve the performance of linear solvers such as MUMPS and HSL. !!! warning - Using HSL linear solvers requires installing Metis. Metis is optional MUMPS. + Using HSL linear solvers requires installing Metis. Metis is optional for MUMPS. -We will install Metis using Anaconda, however, it can also be installed from source. -To install using Anaconda, we will create a clean environment with nothing but Metis. -The conda environment is being constructed within a directory in `hpcapps` project on -Kestrel. +We will install Metis using Anaconda. However, it can also be installed from source. +To install using Anaconda, we will create a clean environment with only Metis. +The conda environment is being constructed within a directory in the `hpcapps` project on +Kestrel. Users can create a conda environment in any place of their choice. ```bash module load conda @@ -49,6 +50,7 @@ conda install conda-forge::metis ``` #### Coinbrew + [Coinbrew](https://github.com/coin-or/coinbrew) is a package manager to install COIN-OR tools. It makes installing IPOPT and its dependencies easier. However, it is not necessary to the installation if one clones the repositories individually. @@ -59,13 +61,21 @@ wget https://raw.githubusercontent.com/coin-or/coinbrew/master/coinbrew ``` #### Intel oneAPI MKL + Intel oneAPI MKL provides BLAS and LAPACK libraries for efficient linear algebra. Additionally, it also provides access to oneMKL PARDISO linear solver that is -compatible with Ipopt. +compatible with IPOPT. !!! note oneMKL PARDISO is not available on Kestrel GPU nodes since they consist of AMD processors. +#### HSL + +[HSL (Harwell Subroutine Library)](http://hsl.rl.ac.uk/ipopt) is a set of linear solvers +that can greatly accelerate the speed of optimization over other linear solvers, e.g., MUMPS. +HSL can be installed separately as well using [ThirdParty-HSL](https://github.com/coin-or-tools/ThirdParty-HSL). +Please see [here](../Development/Libraries/hsl.md) for installation on Kestrel. + ### Installation In this demonstration, we will install Ipopt within @@ -104,6 +114,14 @@ ln -s /projects/hpcapps/kpanda/conda-envs/metis/include/metis.h metis.h cd /projects/msoc/kpanda/apps/ # go back base directory ``` +This has two advantages. +First, we don't need to add `/projects/hpcapps/kpanda/conda-envs/metis/lib/` to +the `LD_LIBRARY_PATH`. The second advantage is that anaconda puts all the +environments libraries and include files in the same directories with +`libmetis.so` and `metis.h`. Many of these libraries overlap with those used +by HSL, Mumps and IPOPT but are not necessarily the same versions. Loading a +different version of a library than those compiled against can cause unexpected behavior. + Next, we will load additional modules. If users require oneMKL PARDISO or would like to leverage intel performance optimization, run the following commands @@ -121,7 +139,8 @@ We will now copy the HSL source code tarball into `/projects/msoc/kpanda/apps/ThirdParty/HSL/`, unpack it, and rename or (create a symbolic link to the unpacked directory) as `coinhsl`. -We are now ready to install Ipopt and its dependencies. Going back to the base +We are now ready to install IPOPT and its dependencies. We will use the default +compilers available in the Kestrel programming environment. Going back to the base directory, we will run the following commands ```bash @@ -147,7 +166,7 @@ export LD_LIBRARY_PATH=/projects/msoc/kpanda/apps/Ipopt/install/lib:${LD_LIBRARY ``` !!! note - Do not forget to load `intel-oneapi-mkl` or `netlib-lapack` before running Ipopt else your runs will fail. + Do not forget to load `intel-oneapi-mkl` or `netlib-lapack` before running IPOPT else your runs will fail. ### Using Custom IPOPT with JuMP @@ -166,4 +185,24 @@ To use our custom installation of IPOPT with `Ipopt.jl`, we do the following: 4. Print the path `Ipopt.jl` has stored for `libipopt.so`. This should be the location of your compiled version. ```julia using Ipopt; println(Ipopt.libipopt_path) - ``` \ No newline at end of file + ``` + +!!! info + The IPOPT build that comes with `Ipopt.jl` seems to expect the HSL library to have the name `libhsl.so`. The repo ThirdParty-HSL builds the library `libcoinhsl.so`. The simplest fix is to do the following: + + ```bash + cd /projects/msoc/kpanda/apps/Ipopt/install/lib # install directory + # Create a symbolic link called libhsl.so + ln -s libcoinhsl.so libhsl.so + ``` + +The following Julia code is useful for testing the HSL linear solvers are working + +```julia +using JuMP, IPOPT + +m = JuMP.Model(()->IPOPT.Optimizer(linear_solver="ma97")) +@variable(m, x) +@objective(m, Min, x^2) +JuMP.optimize!(m) +``` \ No newline at end of file diff --git a/docs/Documentation/Development/Libraries/hsl.md b/docs/Documentation/Development/Libraries/hsl.md index a42f4c48d..cdd43a58e 100644 --- a/docs/Documentation/Development/Libraries/hsl.md +++ b/docs/Documentation/Development/Libraries/hsl.md @@ -2,7 +2,7 @@ title: HSL postdate: October 27, 2020 layout: default -author: Jonathan Maack +author: Jonathan Maack, Kinshuk Panda description: How to install and use HSL with with IPOPT parent: Libraries grand_parent: Development @@ -14,143 +14,135 @@ grand_parent: Development ## Installation -Go to the [HSL for IPOPT](http://www.hsl.rl.ac.uk/ipopt/) site and follow the instructions to request the source code for all the available solvers. Note that the solver MA27 is free to obtain, but MA27 is a serial solver. Other solvers will require a license. Please request a license that applies to your use case. +Go to the [HSL](http://www.hsl.rl.ac.uk/ipopt/) site and follow the instructions to request the source code for all the available solvers. Note that the solver MA27 is free to obtain, but MA27 is a serial solver. Other solvers will require a license. Please request a license that applies to your use case. -### HPC +!!! info + If you are building IPOPT along with HSL, please follow the instructions [here](../../Applications/ipopt.md#installation-from-source). -The default version of IPOPT distributed with `Ipopt.jl` on Linux links to the OpenBLAS library. This causes issues when linking the HSL library to the MKL libraries. For this reason, to use HSL linear solvers with IPOPT on the cluster, either we must compile IPOPT from scratch or compile HSL with OpenBLAS instead of MKL. For performance reasons, we have elected to compile IPOPT from scratch so that we can use the MKL libraries. +We need to be careful regarding the selection of linear algebra libraries when installing HSL. +The default version of IPOPT distributed with `Ipopt.jl` on Linux links to the OpenBLAS library. This causes issues when linking the HSL library to the Intel oneAPI MKL libraries. For this reason, to use HSL linear solvers with IPOPT on Kestrel, either we must compile IPOPT from scratch or compile HSL with OpenBLAS and NetLib LAPACK instead of Intel oneAPI MKL. We demonstrated IPOPT + HSL installation with Intel oneAPI MKL [here](../../Applications/ipopt.md#installation-from-source). -The following provides detailed instructions for compiling IPOPT with HSL and Mumps on the HPC. +The following provides detailed instructions for building HSL using OpenBLAS and Netlib LAPACK ON HPC. #### Pre-requisites ##### Metis -Metis helps the HSL solvers perform better. Therefore, it is recommended that you also install or build the Metis library. If you do want to install Metis, it must be done before compiling the HSL library. +Metis is a serial graph partitioning and fill-reducing matrix ordering software that helps the HSL solvers perform better. Therefore, it is recommended that you also install or build the Metis library. If you do want to install Metis, it must be done before compiling the HSL library. The easiest way to install Metis is to use anaconda: +!!! warning + Using HSL linear solvers requires installing Metis. Metis is optional for MUMPS. + +We will install Metis using Anaconda, however, it can also be installed from source. +To install using Anaconda, we will create a clean environment with nothing but Metis. +The conda environment is being constructed within a directory in `hpcapps` project on +Kestrel. + ```bash module load conda -conda create -n -conda activate -conda install -c conda-forge metis +conda create -p /projects/hpcapps/kpanda/conda-envs/metis python +conda activate /projects/hpcapps/kpanda/conda-envs/metis +conda install conda-forge::metis ``` + !!! info `module load conda` loads the default anaconda module. You may use a different conda module based on your needs. !!! note Anaconda packages sometimes have issues when they come from different channels. We tend to pull everything from `conda-forge` hence the channel choice above. -##### pkg-config - -[pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/) is a helper tool for specifying compiler options while building your code. It is available by default on the HPC. +The Metis library and header files are placed in `/projects/hpcapps/kpanda/conda-envs/metis/lib/` and `/projects/hpcapps/kpanda/conda-envs/metis/include/`, respectively ##### Compilers -We will be using the GNU compiler suite (`gcc` and `gfortran`). These can be accessed on the cluster by loading the appropriate module. This should work with any version of the GNU compilers. We use version 8.4.0 here. These can be loaded by typing `module load gcc/8.4.0`. +We will be using the GNU compiler suite (`gcc` and `gfortran`). These can be accessed on the cluster by loading the appropriate module. This should work with any version of the GNU compilers. We use the default `gcc` and `gfortran` that are available on the CPU compute nodes. #### Setting up the Environment +We will install HSL in `/kfs2/projects/msoc/kpanda/apps/Ipopt/install` for this demonstration. This can be set to whatever location you wish to install. +Let's create the requisite installation directories + +```bash +mkdir -p /kfs2/projects/msoc/kpanda/apps/Ipopt/install +cd /kfs2/projects/msoc/kpanda/apps/Ipopt/install +mkdir lib include +cd .. +``` + We will make use of the following environment variables. ```bash # Location of metis.h -export METIS_HEADER=${HOME}/.conda-envs//include +export METIS_HEADER=/projects/hpcapps/kpanda/conda-envs/metis/include # Location of metis library -export METIS_LIBRARY=${HOME}/.conda-envs//lib +export METIS_LIBRARY=/projects/hpcapps/kpanda/conda-envs/metis/lib # Directory for keeping source code and build products -export MYAPPS=${HOME}/apps +export MYAPPS=/kfs2/projects/msoc/kpanda/apps/Ipopt/install # Location of header files export MYINC=${MYAPPS}/include # Location of static and dynamic libraries export MYLIB=${MYAPPS}/lib ``` -These can be added to the .bash_profile file (or equivalent for other shells). Remember after adding these to source `.bash_profile` (or equivalent) or to open a new terminal and do all building there. -To make the Metis header and dynamic library easily accessible to the HSL, MUMPS and IPOPT libraries, we will put symbolic links in the `${MYINC}` and `${MYLIB}` directories. Do this by doing the following: +These can be added to the `.bash_profile` file (or equivalent for other shells). Remember after adding these to source `.bash_profile` (or equivalent) or to open a new terminal and do all building there. +Alternatively, to make the Metis header and dynamic library easily accessible to the HSL, MUMPS and IPOPT libraries, we will put symbolic links in the `${MYINC}` and `${MYLIB}` directories. Do this by doing the following: ```bash cd ${MYINC} -ln -s ${METIS_HEADER}/metis.h +ln -s ${METIS_HEADER}/metis.h metis.h cd ${MYLIB} -ln -s ${METIS_LIBRARY}/libmetis.so +ln -s ${METIS_LIBRARY}/libmetis.so libmetis.so ``` -This has a couple of advantages. First, the `coinbrew` build will automatically add the `${MYLIB}` directory to the rpath of all constructed libraries and executables. This means that we don't need to add `${MYLIB}` to the LD_LIBRARY_PATH. The other advantage is that anaconda puts all the environments libraries and include files in the same directories with `libmetis.so` and `metis.h`. Many of these libraries overlap with those used by HSL, Mumps and IPOPT but are not necessarily the same versions. Loading a different version of a library than those compiled against can cause unexpected behavior. +This has two advantages. +First, we don't need to add `/projects/hpcapps/kpanda/conda-envs/metis/lib/` to +the `LD_LIBRARY_PATH`. The second advantage is that anaconda puts all the +environments libraries and include files in the same directories with +`libmetis.so` and `metis.h`. Many of these libraries overlap with those used +by HSL, Mumps and IPOPT but are not necessarily the same versions. Loading a +different version of a library than those compiled against can cause unexpected behavior. #### Configure and Install -Follow the [IPOPT installation instructions here](../../Applications/ipopt.md#installation) to finish the installation of HSL solvers on the HPC. - -### MacOS - -The following installation has been tested on Apple's M1 ARM based processors. - -#### Pre-requisites - -We will use [Homebrew](https://brew.sh) and [ThirdParty-HSL](https://github.com/coin-or-tools/ThirdParty-HSL) to install HSL libraries (and IPOPT). As per the [default IPOPT installation instructions](https://coin-or.github.io/Ipopt/INSTALL.html), we will rely on GNU compilers for the installation. Run the following commands +We will clone `ThirdParty-HSL` and configure and install HSL in a working directory ```bash -# Update homebrew and download packages -brew update -brew install bash gcc metis pkg-config -# Create a directory of your choice to install HSL -mkdir -p {$HOME}/UserApps/IPOPT/HSL/hsl_install -cd {$HOME}/UserApps -# Clone ThirdParty-HSL git clone git@github.com:coin-or-tools/ThirdParty-HSL.git -cd ThirdParty-HSL -# Place the HSL source code here -cp -r ${HSL_SOURCE_CODE_LOCATION} coinhsl ``` -#### Setting up the Environment +Copy the HSL source code tarball into `/projects/msoc/kpanda/apps/ThirdParty/HSL/`, +unpack it, and rename or (create a symbolic link to the unpacked directory) as `coinhsl`. -Assuming that you allow Homebrew to install to its default locations, we will declare the following environment variables +Run the following commands to configure ```bash -# Location of metis.h -export METIS_HEADER=/opt/homebrew/Cellar/metis/5.1.0/include -# Location of metis library -export METIS_LIBRARY=/opt/homebrew/Cellar/metis/5.1.0/lib -# Directory for keeping source code and build products -mkdir -p {$HOME}/UserApps/IPOPT/HSL/hsl_install -export MYAPPS={$HOME}/UserApps/IPOPT/HSL/hsl_install -# Location of static and dynamic libraries -mkdir -p ${MYAPPS}/lib -export MYLIB=${MYAPPS}/lib -``` - -#### Configure and Install - -Go to the requisite directory and run the following commands - -```bash -cd {$HOME}/UserApps/ThirdParty-HSL/ -mkdir build && cd build -../configure F77=gfortran-12 FC=gfortran-12 CC=gcc-12 --prefix="${MYAPPS}" \ ---with-metis --with-metis-lflags="-L${METIS_LIBRARY} -lmetis" \ ---with-metis-cflags="-I${METIS_HEADER}" +cd ThirdParty-HSL +module load netlib-lapack +./configure --prefix=${MYAPPS} \ +--with-metis \ +--with-metis-cflags=-I${METIS_HEADER} \ +--with-metis-lflags="-L${METIS_LIBRARY} -lmetis" make && make install ``` -This should install the HSL libraries in `${MYAPPS}`. Finally add `MYLIB` to your `DYLD_LIBRARY_PATH`. You can append the following line to your `.bash_profile` to make it permanent or call it every time you need to run IPOPT with HSL solvers. +This should install the HSL libraries in `${MYAPPS}`. Finally, add `MYLIB` to your `LD_LIBRARY_PATH`. You can append the following line to your `.bash_profile` to make it permanent or call it every time you need to run IPOPT with HSL solvers. ```bash -export export DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:${MYLIB}/UserApps/IPOPT/HSL/hsl_install/lib +export export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MYAPPS}/lib ``` ## Usage IPOPT has a feature called the linear solver loader (read about it [here](https://coin-or.github.io/Ipopt/INSTALL.html#LINEARSOLVERLOADER)). This allows for loading linear solvers from a dynamic library at run time. We will use this feature to use the HSL solvers. -The only thing you have to do is to make the HSL dynamic library findable. This is done by adding the directory containing the HSL library to the environment variable `DYLD_LIBRARY_PATH` in MacOS and `LD_LIBRARY_PATH` on Linux-based systems. See above for MacOS and [here](../../Applications/ipopt.md#using-custom-ipopt-with-jump) for NREL systems. To use the new linear solvers just use the `linear_solver=""` argument to `IPOPT.Optimizer`. +The only thing you have to do is to make the HSL dynamic library findable. This is done by adding the directory containing the HSL library to the environment variable `LD_LIBRARY_PATH`. To use the new linear solvers just use the `linear_solver=""` argument to `IPOPT.Optimizer`. !!! info - The IPOPT build that comes with `Ipopt.jl` seems to expect the HSL library to have the name `libhsl.dylib` on MacOS. The repo ThirdParty-HSL builds the library `libcoinhsl.dylib`. The simplest fix is to do the following: + The IPOPT build that comes with `Ipopt.jl` seems to expect the HSL library to have the name `libhsl.so`. The repo ThirdParty-HSL builds the library `libcoinhsl.so`. The simplest fix is to do the following: ```bash cd ${MYLIB} @@ -158,13 +150,4 @@ The only thing you have to do is to make the HSL dynamic library findable. This ln -s libcoinhsl.dylib libhsl.dylib ``` -The following Julia code is useful for testing the HSL linear solvers are working - -```julia -using JuMP, IPOPT - -m = JuMP.Model(()->IPOPT.Optimizer(linear_solver="ma97")) -@variable(m, x) -@objective(m, Min, x^2) -JuMP.optimize!(m) -``` \ No newline at end of file +Alternatively, users can follow the instructions mentioned [here for Julia JuMP](https://github.com/jump-dev/Ipopt.jl?tab=readme-ov-file#hsl) \ No newline at end of file From 37c50f80ee1e42086dedf1e645e13d0bfdf337c4 Mon Sep 17 00:00:00 2001 From: Kinshuk Panda Date: Wed, 6 Nov 2024 20:06:48 -0700 Subject: [PATCH 3/7] Update docs/Documentation/Applications/ipopt.md Co-authored-by: Haley Yandt <46908710+yandthj@users.noreply.github.com> --- docs/Documentation/Applications/ipopt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Documentation/Applications/ipopt.md b/docs/Documentation/Applications/ipopt.md index 42c5a1dc1..fa6749899 100644 --- a/docs/Documentation/Applications/ipopt.md +++ b/docs/Documentation/Applications/ipopt.md @@ -15,7 +15,7 @@ grand_parent: Development IPOPT is commonly used in solving power flow, e.g., AC Optimal Power Flow, and controls problems. Please refer to their [project website](https://github.com/coin-or/Ipopt) for the source code. The documentation can be found [here](https://coin-or.github.io/Ipopt/index.html). !!! note - IPOPT with HSL linear solvers is available as a module on Kestrel. Please see [IDAES Solvers](./idaes_solvers.md) for additional details. We recommend using the system module for ease-of-use and only build if they do not meet your needs. + IPOPT with HSL linear solvers is available as a module on Kestrel. Please see [IDAES Solvers](./idaes_solvers.md) for additional details. We recommend using the system module for ease-of-use and only build if the module does not meet your needs. ## Installation from source From fe476f3d0df532de1aca0fdf2e14be003504825e Mon Sep 17 00:00:00 2001 From: Kinshuk Panda Date: Wed, 6 Nov 2024 20:07:01 -0700 Subject: [PATCH 4/7] Update docs/Documentation/Applications/ipopt.md Co-authored-by: Haley Yandt <46908710+yandthj@users.noreply.github.com> --- docs/Documentation/Applications/ipopt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Documentation/Applications/ipopt.md b/docs/Documentation/Applications/ipopt.md index fa6749899..718308218 100644 --- a/docs/Documentation/Applications/ipopt.md +++ b/docs/Documentation/Applications/ipopt.md @@ -21,7 +21,7 @@ IPOPT is commonly used in solving power flow, e.g., AC Optimal Power Flow, and c ## Installation from source !!! info - We advise build all applications on the compute node on an interactive session. Please see [Running Interactive Jobs](../Slurm/interactive_jobs.md#running-interactive-jobs) for additional details. + We advise building all applications on a compute node using an interactive session. Please see [Running Interactive Jobs](../Slurm/interactive_jobs.md#running-interactive-jobs) for additional details. ### Optional Pre-requisites From 81ed2adc2316996f32a6296e091c02607a3253a6 Mon Sep 17 00:00:00 2001 From: Kinshuk Panda Date: Wed, 6 Nov 2024 20:07:10 -0700 Subject: [PATCH 5/7] Update docs/Documentation/Applications/ipopt.md Co-authored-by: Haley Yandt <46908710+yandthj@users.noreply.github.com> --- docs/Documentation/Applications/ipopt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Documentation/Applications/ipopt.md b/docs/Documentation/Applications/ipopt.md index 718308218..b969b90a8 100644 --- a/docs/Documentation/Applications/ipopt.md +++ b/docs/Documentation/Applications/ipopt.md @@ -30,7 +30,7 @@ choose depending on their needs. #### Metis -It is highly recommended installing [Metis](https://github.com/KarypisLab/METIS.git) +It is highly recommended to install [Metis](https://github.com/KarypisLab/METIS.git) - Serial Graph Partitioning and Fill-reducing Matrix Ordering software to improve the performance of linear solvers such as MUMPS and HSL. From 7331190df632ed8fd8d339ebb406469f25760cb1 Mon Sep 17 00:00:00 2001 From: Kinshuk Panda Date: Wed, 6 Nov 2024 20:07:22 -0700 Subject: [PATCH 6/7] Update docs/Documentation/Applications/ipopt.md Co-authored-by: Haley Yandt <46908710+yandthj@users.noreply.github.com> --- docs/Documentation/Applications/ipopt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Documentation/Applications/ipopt.md b/docs/Documentation/Applications/ipopt.md index b969b90a8..b77004e2e 100644 --- a/docs/Documentation/Applications/ipopt.md +++ b/docs/Documentation/Applications/ipopt.md @@ -39,7 +39,7 @@ improve the performance of linear solvers such as MUMPS and HSL. We will install Metis using Anaconda. However, it can also be installed from source. To install using Anaconda, we will create a clean environment with only Metis. -The conda environment is being constructed within a directory in the `hpcapps` project on +For this example, the conda environment is being constructed within a directory in the `hpcapps` project on Kestrel. Users can create a conda environment in any place of their choice. ```bash From ff1c44a1fe319739797005e4495804928a114687 Mon Sep 17 00:00:00 2001 From: Kinshuk Panda Date: Wed, 6 Nov 2024 20:07:45 -0700 Subject: [PATCH 7/7] Update docs/Documentation/Development/Libraries/hsl.md Co-authored-by: Haley Yandt <46908710+yandthj@users.noreply.github.com> --- docs/Documentation/Development/Libraries/hsl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Documentation/Development/Libraries/hsl.md b/docs/Documentation/Development/Libraries/hsl.md index cdd43a58e..f3b2701b3 100644 --- a/docs/Documentation/Development/Libraries/hsl.md +++ b/docs/Documentation/Development/Libraries/hsl.md @@ -53,7 +53,7 @@ conda install conda-forge::metis !!! note Anaconda packages sometimes have issues when they come from different channels. We tend to pull everything from `conda-forge` hence the channel choice above. -The Metis library and header files are placed in `/projects/hpcapps/kpanda/conda-envs/metis/lib/` and `/projects/hpcapps/kpanda/conda-envs/metis/include/`, respectively +The Metis library and header files are placed in `/projects/hpcapps/kpanda/conda-envs/metis/lib/` and `/projects/hpcapps/kpanda/conda-envs/metis/include/`, respectively. ##### Compilers