Identify compiler versions during testing

[apcraig]

Automatically track compiler and system (mpi, netcdf, etc) versions during automated testing and report. This would be useful information in the test suites and as part of release information.

[apcraig]

#460 highlights an issue with compiler versions and potential compiler bugs. We should probably start to extend testing to include multiple compiler versions where possible, especially the most newest compilers. This also highlights importance of formally tracking compiler/software versions.

[apcraig]

In the medium term, I'd like to continue to increase the number of machines and compilers we're testing on. I'd also like to find a way to self document the versions, but I have not come up with a good way to do that. In theory, we can query the compiler like "mpif90 --version" and capture the output. We can build that feature into the Makefile or similar. But that will probably not be able to tell us what version of MPI or netcdf we're using. At this point, I'm thinking we might have to hardwire the info into the env file. So for instance, we might have

module purge
module load ncarenv/1.2
module load intel/19.0.2
module load mpt/2.19
module load ncarcompilers/0.5.0
module load netcdf/4.6.3

setenv ICE_MACHINE_ENVNAME cheyenne
setenv ICE_MACHINE_COMPILER intel

and now we'd add

setenv ICE_MACHINE_SOFTWARE "intel/19.0.2, mpt/2.19, netcdf/4.6.3"

We'd just have to remember to update the ICE_MACHINE_SOFTWARE when we change the modules. That doesn't seem too burdensome and gives us the most flexibility. Trying to get the info automatically might be harder. I guess we capture something like "module list" but that often contains a bunch of stuff we don't want to know about. It also wouldn't work for a machine that doesn't have modules on it. Maybe we can start with a hardwired "SOFTWARE" env variable and then automate in the future if we find a good way and need.

Anyone else have any ideas? @phil-blain, @eclare108213

[phil-blain]

What we could do is embed the version number of the different modules in the cice executable itself; this should be robust and we would not have to think about updating anything.

For example, I know the Intel compiler defines a preprocessor macro that uniquely defines the compiler version, so we could use that. Probably other compilers and libraries do the same, I think it's pretty standard. So we could define variables in the model that get initialized at compile time from this macros.

We could then make a small modification to program icemodel that checks if the executable was given an argument, and if yes just prints these variables. So we could then call

$ ./cice --configuration
This is the CICE-Consortium sea-ice model, version <`git --describe` output>
This CICE was compiled with:
  Compiler: <compiler version>
  NetCDF:   <NetCDF version>
  MPI:      <MPI version>
  # etc

See also #353 (for the version number)

[apcraig]

Thanks @phil-blain, interesting idea. Just doing a bit of research

MPI_VERSION and MPI_SUBVERSION seem to be part of the MPI standard, but I'm not convinced that's enough. What we want to know is that MPI is intelmpi or mpt or openmpi or mpich version x.y.z. The MPI version info that MPI provides is also interesting, but I'm not sure is exactly what we want.

NetCDF does seem to provide nf90_inq_libvers which would be useful.

For compilers, I'm a lot less clear we can do something like this. Also, I'm not convinced we should add other modes like "--configuration" for our science codes. But I haven't thought thru that fully yet.

Is there benefit to providing the --configuration to users in general? Or are we just seeking a way to more clearly identify and communicate our test platforms to the community?

Looking a little more into our current situation, the module names will not be adequate documentation either. For instance, on Mustang, I get

>module load compiler/intel/2020.0.1
>ifort --version
ifort (IFORT) 19.1.1.217 20200306
Copyright (C) 1985-2020 Intel Corporation.  All rights reserved.

So we have to be very careful that we actually interrogate the compilers directly to get the proper information. This is becoming more and more tricky. I'm not convinced we can come up with a robust unified approach (ie. query via the model code or leverage the modules list). Would love to be convinced otherwise.

[apcraig]

MACHINFO and ENVINFO hardwired information was added to both Icepack and CICE as an initial implementation. I will leave this open for now in case we need further revisions or advance ways to automate the information collection.