Bugfix: Issues with the met_compile_all.sh script and associated tar files.

[HathewayWill]

@JohnHalleyGotway recommenced this bug fix after discussion on webcall. @jprestop

Authors of bug fix: @HathewayWill

Describe the Problem

• With the update of MET v11 and METplus v5 the compile_MET_all.sh script is broken in a clean environment and missing variable exports.
  -- In compile_MET_all.sh the variable LIB_Z is defined at line 811. However, it is called and referenced earlier in the script causing HDF5 & NETCDF to fail building. While HDF5 will compile and install without zlib, NETCDF requires HDF5 to be built with zlib.
  -- ${LIB_Z} is referenced at line numbers 98, 621, 811, 831.

• Library tar files have been updated and old tar file not archived with version numbers so previous compile_MET_all.sh scripts (ie v10.0.2) will not install properly.
  -- With the update of v11 the netcdf library file in the tar_files.tar changed from a .zip file to a tar.gz file. In the compile_MET_all.sh script from v10.0.2 the script utilizes the old naming structure and compression. Therefore the new tar_files will not work with it.

Expected Behavior

Should be able to run compile_MET_all.sh on a clean environment and install all the required libraries and software packages.

Environment

Describe your runtime environment:
*1. Machine: Linux Workstation
*2. OS: Ubuntu 22.04.1
*3.
-hdf5 1.12.2
-libpng 1.6.37
-zlib 1.2.11
-jasper 1.900.1
-netcdf 4.7.4

To Reproduce

Run the below code in clean environment terminal:

sudo apt -y update
sudo apt -y upgrade
sudo apt -y install python3 python3-dev emacs flex bison libpixman-1-dev libjpeg-dev pkg-config libpng-dev unzip python2 python2-dev python3-pip pipenv gcc gfortran g++ libtool automake autoconf make m4 default-jre default-jdk csh ksh git libncurses5 libncurses6 mlocate pkg-config build-essential curl libcurl4-openssl-dev byacc flex

 pip3 install python-dateutil==2.8

mkdir $HOME/WRF
export WRF_FOLDER=$HOME/WRF
 mkdir $WRF_FOLDER/MET-11.0.0
 mkdir $WRF_FOLDER/MET-11.0.0/Downloads
 mkdir $WRF_FOLDER/METplus-5.0.0
 mkdir $WRF_FOLDER/METplus-5.0.0/Downloads

 cd $WRF_FOLDER/MET-11.0.0/Downloads
 wget -c -4 https://raw.githubusercontent.com/dtcenter/MET/main_v11.0/internal/scripts/installation/compile_MET_all.sh
 wget -c -4 https://dtcenter.ucar.edu/dfiles/code/METplus/MET/installation/tar_files.tgz
 wget -c -4 https://github.com/dtcenter/MET/archive/refs/tags/v11.0.0.tar.gz

 cp compile_MET_all.sh $WRF_FOLDER/MET-11.0.0
 tar -xvzf tar_files.tgz -C $WRF_FOLDER/MET-11.0.0
 cp v11.0.0.tar.gz $WRF_FOLDER/MET-11.0.0/tar_files
 cd $WRF_FOLDER/MET-11.0.0

 # Installation of Model Evaluation Tools
 export CC=gcc
 export CXX=g++
 export FC=gfortran
 export F77=gfortran
 export CFLAGS="-fPIC -fPIE -O3"

 cd $WRF_FOLDER/MET-11.0.0
 export GCC_VERSION=$(/usr/bin/gcc -dumpfullversion | awk '{print$1}')
 export GFORTRAN_VERSION=$(/usr/bin/gfortran -dumpfullversion | awk '{print$1}')
 export GPLUSPLUS_VERSION=$(/usr/bin/g++ -dumpfullversion | awk '{print$1}')

 export GCC_VERSION_MAJOR_VERSION=$(echo $GCC_VERSION | awk -F. '{print $1}')
 export GFORTRAN_VERSION_MAJOR_VERSION=$(echo $GFORTRAN_VERSION | awk -F. '{print $1}')
 export GPLUSPLUS_VERSION_MAJOR_VERSION=$(echo $GPLUSPLUS_VERSION | awk -F. '{print $1}')

 export version_10="10"

 if [ $GCC_VERSION_MAJOR_VERSION -lt $version_10 ] || [ $GFORTRAN_VERSION_MAJOR_VERSION -lt $version_10 ] || [ $GPLUSPLUS_VERSION_MAJOR_VERSION -lt $version_10 ]
 then
   sed -i 's/-fno-second-underscore -fallow-argument-mismatch/-fno-second-underscore -Wno-argument-mismatch/g' compile_MET_all.sh
 fi
 export PYTHON_VERSION=$(/usr/bin/python3 -V 2>&1|awk '{print $2}')
 export PYTHON_VERSION_MAJOR_VERSION=$(echo $PYTHON_VERSION | awk -F. '{print $1}')
 export PYTHON_VERSION_MINOR_VERSION=$(echo $PYTHON_VERSION | awk -F. '{print $2}')
 export PYTHON_VERSION_COMBINED=$PYTHON_VERSION_MAJOR_VERSION.$PYTHON_VERSION_MINOR_VERSION

 export FC=/usr/bin/gfortran
 export F77=/usr/bin/gfortran
 export F90=/usr/bin/gfortran
 export gcc_version=$(gcc -dumpfullversion)
 export TEST_BASE=$WRF_FOLDER/MET-11.0.0
 export COMPILER=gnu_$gcc_version
 export MET_SUBDIR=${TEST_BASE}
 export MET_TARBALL=v11.0.0.tar.gz
 export USE_MODULES=FALSE
 export MET_PYTHON=/usr
 export MET_PYTHON_CC=-I${MET_PYTHON}/include/python${PYTHON_VERSION_COMBINED}
 export MET_PYTHON_LD=-L${MET_PYTHON}/lib/python${PYTHON_VERSION_COMBINED}/config-${PYTHON_VERSION_COMBINED}-x86_64-linux-gnu\ -L${MET_PYTHON}/lib\ -lpython${PYTHON_VERSION_COMBINED}\ -lcrypt\ -lpthread\ -ldl\ -lutil\ -lm
 export SET_D64BIT=FALSE

 chmod 775 compile_MET_all.sh
./compile_MET_all.sh |& tee compile_MET_all.log

These errors occur and we have attached the relevant log files:

compile_MET_all.log
zlib_make.log
zlib_configure.log
zlib_make_install.log
HDF5_config.log
HDF5_configure.log
HDF5_make_install.log
NETCDF_config.log
NETCDF_configure.log

Tar file issue:

Looking at the source link for the tar files (https://dtcenter.ucar.edu/dfiles/code/METplus/MET/installation/) the files were updated on 2022-12-21 17:42 and there is no archive for the old files.

Comparing the two compile_MET_all files (below) and looking at the commits in the change log ( 21d54bf) you can see that compile_MET_all.sh has changed. The previous version of compile_MET_all.sh (v10.0.2) will not work with the new tar file package.
compile_MET_all_v10.0.2.sh.txt
compile_MET_all_v11.0.0.sh.txt

In the tar file package the version numbers are:

Suggested Fixes

• For compile_MET_all.sh there are two options to fix the issue:
  -- In the netcdf compile (Line 621) , export LIB_Z is not defined; changing it to ${LIB_DIR}/lib
  -- In the zlib compile (Line 425) add new line at the bottom of the if statement with export LIB_Z=${LIB_DIR}/lib

• For the tar file packages
  -- Create a separate tar file package for v10.0.2

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Organization level Project for support of the current coordinated release
• Select Repository level Project for development toward the next official release or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Bugfix Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of main_<Version>.
  Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
• Fix the bug and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into main_<Version>.
  Pull request: bugfix <Issue Number> main_<Version> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Organization level software support Project for the current coordinated release
  Select: Milestone as the next bugfix version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Complete the steps above to fix the bug on the develop branch.
  Branch name: bugfix_<Issue Number>_develop_<Description>
  Pull request: bugfix <Issue Number> develop <Description>
  Select: Reviewer(s) and Development issue
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Close this issue.

[jprestop]

@JohnHalleyGotway Thank you for having a web call to discuss these issues.

For compile_MET_all.sh there are two options to fix the issue:
-- In the netcdf compile (Line 621) , export LIB_Z is not defined; changing it to ${LIB_DIR}/lib
-- In the zlib compile (Line 425) add new line at the bottom of the if statement with export LIB_Z=${LIB_DIR}/lib
For the tar file packages
-- Create a separate tar file package for v10.0.2

Note to self about preferred methods so I don't forget when I get back:

• Set LIB_Z to LIB_DIR if LIB_Z is not already set as some machines will not use LIB_DIR for LIB_Z.
• Since the tar file package is not in version control and is referenced on this webpage, instead of creating a separate tar file package for main_v10.1, I propose updating the compile_MET_all.sh script to read the new NetCDF tar.gz file instead of the older versioned NetCDF .zip file.

@JohnHalleyGotway Please let me know if you foresee any problems with these methods.

[HathewayWill]

I propose updating the compile_MET_all.sh script to read the new NetCDF tar.gz file instead of the older versioned NetCDF .zip file.

You are referring to the compile_MET_all.sh script for v10.1 correct? As compile_MET_all.sh v11.0 already has tar.gz

Set LIB_Z to LIB_DIR if LIB_Z is not already set as some machines will not use LIB_DIR for LIB_Z.

Would this be a IF or CASE statement?

Otherwise these changes sound good to me.

[jprestop]

@HathewayWill
Yes, I was referring to the compile_MET_all.sh script for the code in the main_v10.1 branch, which is for version 10.1.

Would this be a IF or CASE statement?

I haven't looked at the script or details, since I'm out, but it would be something like that.

[georgemccabe]

Hi All,

I ran into the same issue re: zlib, so I created a branch bugfix_2390_develop_compile_zlib with some changes that resolved things for me. I also fixed a few other small typos. If we are happy with these changes we could apply them as a bugfix to main_v11.0 as well.

For the change of the NetCDF4 file from zip to tar.gz, I propose that we add a check for that file and use the appropriate (un)compression software depending on which file is there. This way someone who has already obtained the old tar_files.tgz file can still install a bugfix release for 10.1. Perhaps we could write a helper function to handle this step for each component in case the file type changes in the future?

I do agree that it would be nice to have some sort of version control for the tar files so we can clearly see when the components have changed. We can discuss options in the next METplus meeting. At very least adding a version number to the tar_files.tgz filename that will be incremented each time it is changed and a README describing what has changed for each version would be helpful to know what is being used.