Create an Amazon AMI from scratch and store the commands for doing so in the METplus repo.

[JohnHalleyGotway]

Describe the Task

This task is very similar to METplus #506. However for METplus #506, we started with the existing METviewer AMI from NOAA and added the other METplus components to it. The goal for this issue is to create a script to construct a METplus AMI totally from scratch, without starting from the METviewer NOAA AMI.

The majority of the work should already be done for #506 and can be incorporated into an AMI build script. Recommend storing the resulting METplus build AMI script somewhere in the METplus repo, but I don't know exactly where yet.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the task down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
• Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Task Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s), Project(s), Milestone, and Linked issues
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[hankenstein2]

sudo apt install git
sudo apt install python3
sudo apt install python3-dev
sudo apt install gcc
sudo apt install gfortran
sudo apt install make
sudo apt install unzip
sudo apt install g++
sudo apt install m4
sudo apt install build-essential
sudo apt install libc++-dev
sudo apt install emacs
sudo apt install python3-pip
sudo apt install libhdf5-dev
sudo apt install libnetcdf-dev
sudo pip3 install netCDF4
sudo pip3 install numpy
sudo pip3 install xarray - installs pandas - needs min 4gb memory
sudo pip3 install argparse

[hankenstein2]

MET install as well
wget https://raw.githubusercontent.com/dtcenter/MET/develop/scripts/installation/compile_MET_all.sh
chmod 775 compile_MET_all.sh

wget https://dtcenter.ucar.edu/dfiles/code/METplus/MET/installation/tar_files.tgz
tar -zxf tar_files.tgz

wget https://github.com/dtcenter/MET/releases/download/v10.0.1/met-10.0.1.20211201.tar.gz

[ian-noaa]

Thanks, Hank! A couple of questions:

• Are there any version constraints on the Debian dependencies you installed? I'm suspecting I'll be handed a CentOS 7 base instance.
• How did you configure the environment for the compile_MET_all.sh script?
• Did you need to do any installation with the tar_files.tgz and met-10.0.1.20211201.tar.gz tarballs?
• Was there any other configuration required for the other parts of the METplus suite (METplus wrappers, METCalcPy/PlotPy, METdatadb), or are those installed by compile_MET_all.sh?

[hankenstein2]

No version constraints that I remember Configuration below the compile_MET_all.sh script will handel the tar files. You need to put all zipped tar files and the met tar file into the directory tar_files. Should be some explanations with the link I sent. git clone https://github.com/dtcenter/METplus cd METplus manage_externals/checkout_externals -e build_components/Externals.cfg This should put all the METplus components into the same level as METplus You will have to customize your cflags and ldflags by running python3.7-config --cflags python3.7-config --ldflags or whatever version of python you have on the system ###### Environment examples for a non-module based system ##### export MET_TARBALL=met-10.0.0.20210510.tar.gz export TEST_BASE=/opt/MET export COMPILER=gnu_8.3.0 export MET_SUBDIR=${TEST_BASE} export MET_PYTHON_CC=-I/usr/include/python3.7m export MET_PYTHON_LD=-L/usr/lib/python3.7/config-3.7m-x86_64-linux-gnu\ -L/usr/lib\ -lpython3.7m\ -lcrypt\ -lpthread\ -l dl\ -lutil\ -lm\ -Xlinker\ -export-dynamic #### Python version on your system export PYTHON_MODULE=python_3.7.3 ### Local system ${PYTHON_MODULE} install location for python libraries export MET_PYTHON=/usr export USE_MODULES=FALSE export PATH=/usr/bin:/usr/local/bin:/bin export EXTERNAL_LIBS=/opt/METplus/build_components/external_libs export MET_NETCDF=${EXTERNAL_LIBS}/lib export MET_HDF5=${EXTERNAL_LIBS}/lib export MET_BUFRLIB=${EXTERNAL_LIBS}/lib export MET_GRIB2CLIB=${EXTERNAL_LIBS}/lib export MET_GRIB2CINC=${EXTERNAL_LIBS}/include export MET_GSL=${EXTERNAL_LIBS}/lib export BUFRLIB_NAME=-lbufr export GRIB2CLIB_NAME=-lgrib2c export MET_GSL=${EXTERNAL_LIBS}/lib export BUFRLIB_NAME=-lbufr export GRIB2CLIB_NAME=-lgrib2c export LIB_JASPER=${EXTERNAL_LIBS}/lib export LIB_LIBPNG=${EXTERNAL_LIBS}/lib export LIB_Z=${EXTERNAL_LIBS}/lib export SET_D64BIT=FALSE

…

On Wed, Feb 23, 2022 at 5:05 PM Ian McGinnis ***@***.***> wrote: Thanks, Hank! A couple of questions: - Are there any version constraints on the Debian dependencies you installed? I'm suspecting I'll be handed a CentOS 7 base instance. - How did you configure the environment for the compile_MET_all.sh script? - Did you need to do any installation with the tar_files.tgz and met-10.0.1.20211201.tar.gz tarballs? - Was there any other configuration required for the other parts of the METplus suite (METplus wrappers, METCalcPy/PlotPy, METdatadb), or are those installed by compile_MET_all.sh? — Reply to this email directly, view it on GitHub <#859 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHAUIKCCNHT34J4I6D2HVBDU4VY5NANCNFSM42CQM6XQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were assigned.Message ID: ***@***.***>

-- -------------------------------------------------------------- Hank Fisher Research Applications Laboratory 303-497-2817 National Center for Atmospheric Research ***@***.*** P.O. Box 3000, Boulder CO 80307-3000 --------------------------------------------------------------

[jprestop]

@fisherhucar It would be great if you could check in your configuration file (perhaps install_met_env.aws?) at this location:
https://github.com/dtcenter/MET/tree/develop/scripts/installation/config

[ian-noaa]

I've been referencing the various MET & METplus install docs as well as the MET Dockerfile which was quite helpful and have successfully installed MET on a CentOS 7 AMI. However, I had some additional questions after doing so.

Thus far I've:

1. Installed MET dependencies listed in the MET Dockerfile & listed by Hank above
2. Downloaded & compiled MET with compile_MET_all.sh and a custom env file that sets TEST_BASE, COMPILER, MET_SUBDIR, MET_TARBALL, USE_MODULES, PYTHON_LOG, MET_PYTHON, and MET_PYTHON_{LD,CC}
3. Cloned METplus & then called manage_externals/checkout_externals -e build_components/Externals.cfg

This has lead to a few questions:

1. It looks like METplus's checkout_externals also can build MET. So did I need to do step 2? Or did I do this out of order?
2. Additionally, does METplus need anything set or persisted in the environment so that it can find the MET binaries?
3. Is there a traditional installation location for METplus? Based on the environment Hank provided above, I'm guessing the default is to install everything in /opt. Is that correct?

[hankenstein2]

1. You did need to do step 2. The automatic MET build found in build_components is brittle (i.e. only works for some specific cases) 2. Generally having <path>met/bin in your path is what you need. Many of the use cases have environment variables you need to set in order for them to work but they are different and specific to each use case 3. Traditional is a good word, maybe 60% of our installs go in /opt but it really depends on the sys-admin in each case. Hank

…

On Mon, Mar 7, 2022 at 4:45 PM Ian McGinnis ***@***.***> wrote: I've been referencing the various MET <https://met.readthedocs.io/en/latest/Users_Guide/installation.html> & METplus <https://metplus.readthedocs.io/en/main_v4.0/Users_Guide/installation.html> install docs as well as the MET Dockerfile <https://github.com/dtcenter/MET/blob/main_v10.0/scripts/docker/Dockerfile> which was quite helpful and have successfully installed MET on a CentOS 7 AMI. However, I had some additional questions after doing so. Thus far I've: 1. Installed MET dependencies listed in the MET Dockerfile & listed by Hank above 2. Downloaded & compiled MET with compile_MET_all.sh and a custom env file that sets TEST_BASE, COMPILER, MET_SUBDIR, MET_TARBALL, USE_MODULES, PYTHON_LOG, MET_PYTHON, and MET_PYTHON_{LD,CC} 3. Cloned METplus & then called manage_externals/checkout_externals -e build_components/Externals.cfg This has lead to a few questions: 1. It looks like METplus's checkout_externals also can build MET. So did I need to do step 2? Or did I do this out of order? 2. Additionally, does METplus need anything set or persisted in the environment so that it can find the MET binaries? 3. Is there a traditional installation location for METplus? Based on the environment Hank provided above, I'm guessing the default is to install everything in /opt. Is that correct? — Reply to this email directly, view it on GitHub <#859 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHAUIKD3VOBFUD3EYEGW3VLU62IJJANCNFSM42CQM6XQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- -------------------------------------------------------------- Hank Fisher Research Applications Laboratory 303-497-2817 National Center for Atmospheric Research ***@***.*** P.O. Box 3000, Boulder CO 80307-3000 --------------------------------------------------------------

[georgemccabe]

A few follow up notes:

• There is an open issue Fix build components scripts to install MET and other METplus components #1068 to improve the build components script. This will require Restructure the contents of the MET repository so that it matches the existing release tarfiles. MET#1920 to be completed. Currently you cannot build MET from cloning the repository. You need to obtain a release tar file and build from that.
• To configure METplus to find the MET binaries, you will need to set the METplus config variable MET_INSTALL_DIR either in the defaults.conf file or in a user-defined conf file. The former is likely what you will need here. See https://metplus.readthedocs.io/en/develop/Users_Guide/systemconfiguration.html#default-configuration-file for more information. The METplus online tutorial instructs users to set environment variables that are referenced both in the METplus conf files and in the tutorial instructions, but the setting the environment variables alone will not configure METplus to find the MET tools.

[ian-noaa]

Thanks George & Hank,

Regarding #1068 - I had noticed the issue with Externals.cfg as well. I worked around it by changing the branch lines to tag lines. Would it be better to patch Externals.cfg with the correct branch name? Likewise, if MET isn't ready for the build_components script - can I just remove the MET section from the Externals.cfg file so that it skips that install?

Thanks for the tip on the config files. Setting things up in defaults.conf does sound like the way to go.

Are the release tarballs different from the git repo contents for METplus? Since this AMI will be used for work on METplus I'll need the METplus git repo's develop branch checked out on the AMI.

[georgemccabe]

I did not know that tag was a valid key in Externals.cfg. That is good to know! With that knowledge I'll have to think about if it is better to use branches or tags here. Using the main_vX.Y branch will obtain any bugfixes, which is nice. I wasn't aware that the manage externals logic could install. I thought that it only obtained the code from the repositories.

The release tar files pertain to MET and can be found here: https://dtcenter.org/community-code/metplus/download
Here is an example: https://github.com/dtcenter/MET/releases/download/v10.0.1/met-10.0.1.20211201.tar.gz
You have to use one of these tar files to install the MET executables. A script is run to prepare the contents of the MET Git repo for a release.

You can use the METplus develop branch, but I would warn you that it is risky to do so since that branch is constantly changing with new development. It is also more difficult to tell the state of the code using develop since the version number output in the log files does not narrow down exactly which commit was used. Using a stable version of METplus is safer and makes it easier to debug issues because the exact state of the code can be easily obtained or examined.