Future feature for all MET systems

[HathewayWill]

Good morning DTC,

I wanted to start a discussion for the next major release of all the MET systems.

Having used the met software systems and watching my system resource monitor, I have noticed that the software is built using either -j 2 or -j 1.

This takes a very long time for configuring and compiling on computers. I was wondering in future builds would it be feasible to add the full -j options for parallel implementation?

For example, WRF and WPS for ./compile allows the user to input a specific -j (n) value. Which can help speed up the process of installation.

  if ( ! $?WRF_SRC_ROOT_DIR ) setenv WRF_SRC_ROOT_DIR `pwd`

# new dec 2009.  check to see if make supports parallel -j option
  make -j 2 >& /dev/null
  if ( $status == 0 ) then              # Parallel make ok
    if ( ! $?J ) then                   # J not defined
      echo setting parallel make -j 2   # Set default to 2
      setenv J "-j 2"
    else
#     J is defined, check that it is a correctly formed variable
      set first2chars = `echo $J | cut -c 1-2`         # Are 1st two chars are -j?
      set second_word = `echo $J | cut -d" " -f2`      # Is second word a number?
      if    ( "$first2chars" == "-j" ) then
         if ( ( "$second_word" >= "2"  )  && \
              ( "$second_word" <= "20" ) ) then
            echo setting parallel make $J
         else if ( "$second_word" == "1" ) then
            echo setting serial make $J
         else
            echo "badly formed -j option for parallel make: $J"
            echo "or you set the number of processors above 20 "
            echo setting parallel make -j 2            # Set default to 2
            setenv J "-j 2"
         endif
      else
         if ( "$J" == "" ) then     # J blank is OK
            echo setting serial make $J
         else
            echo "parallel option for make is -j, you entered: $first2chars"
            echo setting parallel make -j 2               # Set default to 2
            setenv J "-j 2"
         endif
      endif
    endif
  else
    echo not setting parallel make
  endif

  /bin/rm -f $ZAP >& /dev/null
  make $arglist A2DCASE="$A2DCASE" WRF_SRC_ROOT_DIR="$WRF_SRC_ROOT_DIR"

endif

I hope you consider my suggestion.

Regards,
WH & MR

[georgemccabe]

Hi Will,

Thank you for this suggestion. Are you suggesting that we allow the -j make option for compiling all of the external libraries as well or just for MET itself?

I did some testing and when I added -j to all of the make commands in the compile script, the MET code failed to build because it couldn't find some g2c libraries, but it wasn't immediately clear why this happened since each make step appeared to run successfully. I tried installing all of the libs without -j, then MET with and without -j, which was successful and cut the compile time for MET by almost half using -j.

[HathewayWill]

Hi Will,

Thank you for this suggestion. Are you suggesting that we allow the -j make option for compiling all of the external libraries as well or just for MET itself?

I did some testing and when I added -j to all of the make commands in the compile script, the MET code failed to build because it couldn't find some g2c libraries, but it wasn't immediately clear why this happened since each make step appeared to run successfully. I tried installing all of the libs without -j, then MET with and without -j, which was successful and cut the compile time for MET by almost half using -j.

Good afternoon @georgemccabe @HathewayWill ,

So I'm suggesting that -j be used for the libraries and MET compilation.

I did a little digging and found that MET was created in the late 2000's early 2010's and back then home desktops really didn't have the power and processors to make it viable for parallelization installation.

Now that most computers are multi-core multi-thread I feel that a -j option for MET would go over well.

As you mentioned the install time can get cut by at least half with these parallelization.

I'd be curious to see what changes you made to the compile scripts to make them run in parallel.

Image is of the processors being used when met is being compiled without -j

[georgemccabe]

Hi @HathewayWill, I am sorry for the delayed response. To test using the -j option, I added a variable to the compile script that I added after each make command. I set this variable to -j without an integer, which does not limit the number of jobs used. When I added the option to every make command in the script, it appeared to build all of the libraries properly, but failed to build MET. I then removed the files generated from the script and reran setting -j for the MET make steps but not for any of the external libraries. This gave me a successful build. I have been pulled away to other work, but I was planning on running some additional tests to try to narrow down which library is not happy with the -j option.

Could you tell me a little more about the script excerpt you shared? Is the first make -j 2 >& /dev/null command run to check if the -j option is supported in the version of make used to compile? If so, where is this command run to perform a quick check without compiling all of the code?

[HathewayWill]

Hi @georgemccabe

I'm pretty sure that code came out of the WRF configure file. I'll double check.

As for which library doesn't like -j. Ill dig around too an see which one it is.

I've installed most of those libraries before so it shouldn't be to hard to find it.

[georgemccabe]

Hi @HathewayWill, since the build errors in MET were related to g2clib, I first tried using the -j argument for all make commands except for that library. This appeared to work! I propose that if the additional make arguments are added by the user, then do not apply them to the g2clib compilation. We will want to compile with this option in our automated testing environment to make sure that all of the tests pass, as it could compile successfully but produce unexpected results.

[HathewayWill]

Hi @georgemccabe

I can confirm too that it was the g2clib that didn't work with -j.

I just got done running all the other library installs with -j 32 (my max) and it decreased install time from 25min to almost 10mins.

I didn't know where to add the -j command to the met compile itself. My guess is once that is added too, install times on my machine could be under 5mins.

[georgemccabe]

Support for setting -j for the make commands called from the compile script (except for g2clib) is now available in the develop branch from this pull request. The environment variable MAKE_ARGS can now be used to set -j.

To run make with 16 jobs:

export MAKE_ARGS="-j 16"

To run make with unlimited jobs:

export MAKE_ARGS=-j

[HathewayWill]

This is a great development step for MET!!!