-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bump Python to 3.11.7 #1217
Bump Python to 3.11.7 #1217
Conversation
When testing on S4, I noticed skylab referencing ecflow's python3.9 |
Thanks for testing this @ashley314 . The ecflow server was built as an external package (because of Intel) with an external Python. This is not going to change unless it breaks with the update to Python 3.11.7 (I sincerely hope not). This year, we'll all need to move away from the classic Intel compilers (at least the C/C++ compilers) and use the new LLVM-based compilers ( |
…eature/python_3p11p7
…kage and in py-xnrl about the +delayed variant
…eature/python_3p11p7
…eature/python_3p11p7
…eature/python_3p11p7
…spack-stack into feature/python_3p11p7
@climbfuji One issue I see for EMC_verif-global is that MET and METplus are version 11.1.1 and 5.1.0, respectively, while we need version 9.1.3 and 3.1.1. I will try out a test installation with both pairs of MET and METplus. If successful, should I open a PR to your branch or open a new PR? |
A new PR please. It would be really nice if we could use one somewhat recent set of MET and METplus, though! |
I've seen the same on an AWS instance that I'm testing on. After the oops PR got merged, no more issues with nlohmann-json packages. |
@climbfuji @ulmononian I receive the following errors when attempting to build the UFS from within the global-workflow (from > ./tests/compile.sh hera '-DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8_ugwpv1,FV3_GFS_v17_coupled_p8_ugwpv1,FV3_global_nest_v1 -DPDLIB=ON' 0 intel YES NO
...
-- Found PTSCOTCHparmetis: /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/scotch-7.0.4-c2fm4mk/lib/libptscotchparmetisv3.a
Building MOM6 standalone executable
-- Configuring done (52.8s)
CMake Error at /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/MAPL-targets.cmake:70 (set_target_properties):
The link interface of target "MAPL_cfio_r4" contains:
ESMF::ESMF
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/mapl-config.cmake:74 (include)
CMake Error at /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/MAPL-targets.cmake:122 (set_target_properties):
The link interface of target "MAPL.field_utils" contains:
ESMF::ESMF
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/mapl-config.cmake:74 (include)
GOCART/CMakeLists.txt:70 (find_package)
CMake Error at /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/MAPL-targets.cmake:153 (set_target_properties):
The link interface of target "MAPL.base" contains:
ESMF::ESMF
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/mapl-config.cmake:74 (include)
GOCART/CMakeLists.txt:70 (find_package)
CMake Error at /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/MAPL-targets.cmake:164 (set_target_properties):
The link interface of target "MAPL" contains:
ESMF::ESMF
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/mapl-config.cmake:74 (include)
GOCART/CMakeLists.txt:70 (find_package)
CMake Error at /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/MAPL-targets.cmake:180 (set_target_properties):
The link interface of target "MAPL.cap" contains:
ESMF::ESMF
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/mapl-config.cmake:74 (include)
GOCART/CMakeLists.txt:70 (find_package)
CMake Error at /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/MAPL-targets.cmake:188 (set_target_properties):
The link interface of target "MAPL.history" contains:
ESMF::ESMF
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/mapl-config.cmake:74 (include)
GOCART/CMakeLists.txt:70 (find_package)
CMake Error at /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/MAPL-targets.cmake:196 (set_target_properties):
The link interface of target "MAPL.orbit" contains:
ESMF::ESMF
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/mapl-config.cmake:74 (include)
GOCART/CMakeLists.txt:70 (find_package)
CMake Error at /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/MAPL-targets.cmake:204 (set_target_properties):
The link interface of target "MAPL.ExtData" contains:
ESMF::ESMF
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/mapl-config.cmake:74 (include)
GOCART/CMakeLists.txt:70 (find_package)
CMake Error at /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/MAPL-targets.cmake:212 (set_target_properties):
The link interface of target "MAPL.ExtData2G" contains:
ESMF::ESMF
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/mapl-config.cmake:74 (include)
GOCART/CMakeLists.txt:70 (find_package)
CMake Error at /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/MAPL-targets.cmake:220 (set_target_properties):
The link interface of target "MAPL.griddedio" contains:
ESMF::ESMF
but the target was not found. Possible reasons include:
* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
Call Stack (most recent call first):
/scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/intel/2021.5.0/mapl-2.46.2-anh2zzb/lib64/cmake/MAPL/mapl-config.cmake:74 (include)
GOCART/CMakeLists.txt:70 (find_package)
-- Generating done (1.3s)
CMake Generate step failed. Build files cannot be regenerated correctly. Here are the changes I made to the local ufs_modules = {
{["jasper"] = "2.0.32"},
- {["zlib"] = "1.2.13"},
+ {["zlib-ng"] = "2.1.6"},
{["libpng"] = "1.6.37"},
- {["hdf5"] = "1.14.0"},
+ {["hdf5"] = "1.14.3"},
{["netcdf-c"] = "4.9.2"},
{["netcdf-fortran"] = "4.6.1"},
- {["parallelio"] = "2.5.10"},
- {["esmf"] = "8.6.0"},
+ {["parallelio"] = "2.6.2"},
+ {["esmf"] = "8.6.1"},
{["fms"] = "2023.04"},
{["bacio"] = "2.4.1"},
- {["crtm"] = "2.4.0"},
- {["g2"] = "3.4.5"},
+ {["crtm"] = "2.4.0.1"},
+ {["g2"] = "3.4.9"},
{["g2tmpl"] = "1.10.2"},
- {["ip"] = "4.3.0"},
+ {["ip"] = "5.0.0"},
{["sp"] = "2.5.0"},
{["w3emc"] = "2.10.0"},
- {["gftl-shared"] = "1.6.1"},
- {["mapl"] = "2.40.3-esmf-8.6.0"},
+ {["gftl-shared"] = "1.9.0"},
+ {["mapl"] = "2.46.2-esmf-8.6.1"},
{["scotch"] = "7.0.4"},
} |
@DavidHuber-NOAA |
There is a veeeeery long thread in the ufs-weather-model repo regarding this problem. Fixing this requires mapl 2.46.3 instead of 2.46.2 (we already made this update in spack-stack develop), and also an update to GOCART (outside of spack-stack). Maybe some more code changes in the ufs-weather-model itself. See ufs-community/ufs-weather-model#2399 as an entry point. |
Great, thanks for the heads up @RatkoVasic-NOAA @climbfuji. |
@DavidHuber-NOAA @climbfuji @ulmononian
It ran and passed ATM tests
I used Dave's ufs_hera.intel.lua and ufs_common.lua from Hera: /scratch1/NCEPDEV/global/David.Huber/GW/gw_python_3117/sorc/ufs_model.fd/modulefiles |
All global-workflow submodules build with the exception of gfs-utils. There appears to be a duplicate skgb.f module in g2 v3.4.9 and w3emc v2.10.0. I'm guessing the CMakeLists.txt file will need to be updated for gfs-utils. Opened issue NOAA-EMC/gfs-utils#76 to track. |
Easy fix. Just had to update the CMakeLists files for two executables. |
All submodules of the global-workflow built successfully 🎉 |
@
this is great news. thanks @RatkoVasic-NOAA. just curious, where are your test dirs so i can take a look? |
All GSI regression tests passed except |
@ulmononian : |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested:
#1217 (comment)
Approved
Not a reviewer, so cannot approve the PR; able to duplicate @RatkoVasic-NOAA's results. |
@rickgrubin-noaa Invited you as collaborator for both spack-stack and spack. |
@malloryprow indicated that the version of matplotlib is not compatible with verif-global and could not generate a plot with it. Has this version of matplotlib been tested by others? |
Not by us (NRL). I suppose it's the same situation as for JCSDA (and NRL's downstream plotting tools). Once the update is in, we need to go and fix our old plotting scripts? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this, I'll patch this into the docker PR and rebuild.
@climbfuji I'm not sure. According to her post (NOAA-EMC/EMC_verif-global#118 (comment)), she could not create a simple example plot (https://matplotlib.org/stable/gallery/lines_bars_and_markers/simple_plot.html). Hopefully we can work through the issue during the release candidate period. |
* jcsda/develop: Bump Python to 3.11.7 (JCSDA#1217) Add a clause in the cleanup to fix directory permissions (JCSDA#1273) Bug fix: configure neptune-env variants in three templates: neptune-dev, skylab-dev, unified-dev (JCSDA#1268) Update site configs for Atlantis, Narwhal, Nautilus (JCSDA#1266) Configuration for bufr_query@0.0.2 and eckit@1.27.0 (JCSDA#1240)
I am pretty sure that the matplotlib developers tested their release before they put it out there. Either it's a problem with the version combinations of the different packages we are using, or with the developer's code. Either way we'll need to figure this out during the release candidate period, as you said. |
Summary
Bump
python
to 3.11.7Testing
Applications affected
All
Systems affected
All
Dependencies
Issue(s) addressed
Resolves #1038
Checklist