Numpy version update #1164

lee1043 · 2024-10-29T04:59:41Z

This is one of steps toward xCDAT migration from CDAT.

numpy version has been fixed as 1.23.5 because of conflict with CDAT. Patch of CDAT particular for this conflict was made, which will work with numpy >= 2.0 that is considered for the next version of E3SM Unified environment.

PMP's step toward entire migration:

Update the numpy dependency and make sure it won't break anything.
Remove CDAT dependency.

This PR is for the above step 1.

Demo notebook run check:

lee1043 · 2024-10-29T21:52:54Z

Conflict with xcdat: https://github.com/xCDAT/xcdat/blob/96bc649ad59750ec37b959d9d4890891b54c62c7/conda-env/dev.yml#L14.

show details

$ conda env create -n pmp_devel_20241029 -f conda-env/dev.yml 
/Users/lee1043/mambaforge/lib/python3.10/argparse.py:1936: FutureWarning: `remote_definition` is deprecated and will be removed in 25.9. Use `conda env create --file=URL` instead.
  action(self, namespace, argument_values, option_string)
Channels:
 - conda-forge
 - defaults
Platform: osx-64
Collecting package metadata (repodata.json): done
Solving environment: failed
Channels:
 - conda-forge
 - defaults
Platform: osx-64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides _numpy_rc needed by numpy-2.0.0rc1-py310h4f3048d_0

Could not solve for environment specs
The following packages are incompatible
├─ numpy >=2.0  is installable with the potential options
│  ├─ numpy [2.0.0rc1|2.0.0rc2|2.1.0rc1] would require
│  │  └─ _numpy_rc, which does not exist (perhaps a missing channel);
│  └─ numpy [2.0.0|2.0.1|...|2.1.2], which can be installed;
└─ xcdat >=0.7.2  is not installable because it requires
   └─ numpy >=1.23.0,<2.0  but there are no viable options
      ├─ numpy [1.23.0|1.23.1|...|1.26.4] conflicts with any installable versions previously reported;
      └─ numpy [2.0.0rc1|2.0.0rc2|2.1.0rc1], which cannot be installed (as previously explained).

@tomvothecoder do we have any plan to update numpy version dependency of xcdat?

tomvothecoder · 2024-10-29T21:55:35Z

Just replied here, yes I will remove the numpy <2.0.0 constraint soon.

lee1043 · 2024-10-29T21:56:48Z

Just replied here, yes I will remove the numpy <2.0.0 constraint soon.

Good to know, thank you!

tomvothecoder · 2024-10-31T22:37:34Z

@lee1043 I will release xCDAT v0.7.3 with the new numpy constraints (same as this PR) next week.

lee1043 · 2024-11-06T21:00:11Z

xcdat v0.7.3 released: https://github.com/xCDAT/xcdat/releases/tag/v0.7.3

tomvothecoder · 2024-11-06T21:20:56Z

xcdat v0.7.3 is now available on conda-forge: https://anaconda.org/conda-forge/xcdat

…lict with updated numpy version

lee1043 · 2024-11-06T22:47:28Z

Now I can create conda env from dev.yml without any conflict

show details

conda env create -n pmp_devel_20241106_xcdat0.7.3 -f conda-env/dev.yml

/Users/lee1043/mambaforge/lib/python3.10/argparse.py:1936: FutureWarning: `remote_definition` is deprecated and will be removed in 25.9. Use `conda env create --file=URL` instead.
  action(self, namespace, argument_values, option_string)
Channels:
 - conda-forge
 - defaults
Platform: osx-64
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages:
                                                                                                                                                                                                                     
Preparing transaction: done                                                                                                                                                                                          
Verifying transaction: done                                                                                                                                                                                          
Executing transaction: - Enabling nb_conda_kernels...                                                                                                                                                                
CONDA_PREFIX: /Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3                                                                                                                                           
Status: enabled                                                                                                                                                                                                      
                                                                                                                                                                                                                     
| Enabling notebook extension nb_conda/main...                                                                                                                                                                       
      - Validating: OK                                                                                                                                                                                               
Enabling tree extension nb_conda/tree...                                                                                                                                                                             
      - Validating: OK                                                                                                                                                                                               
Enabling: nb_conda                                                                                                                                                                                                   
- Writing config: /Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/etc/jupyter                                                                                                                           
    - Validating...                                                                                                                                                                                                  
      nb_conda 2.2.1 OK                                                                                                                                                                                              
                                                                                                                                                                                                                     
done                                                                                                                                                                                                                 
#                                                                                                                                                                                                                    
# To activate this environment, use                                                                                                                                                                                  
#                                                                                                                                                                                                                    
#     $ conda activate pmp_devel_20241106_xcdat0.7.3                                                                                                                                                                 
#                                                                                                                                                                                                                    
# To deactivate an active environment, use                                                                                                                                                                           
#
#     $ conda deactivate

lee1043 · 2024-11-06T22:53:58Z

Error from Demo 1a notebook

Click here for Error message

%%bash
pcmdi_compute_climatologies.py -p basic_annual_cycle_param.py

start and end are  2003-01   2018-12
variable list:  ['rlut']
ver: v20241023
var: rlut
infile: demo_data_tmp/obs4MIPs_PCMDI_monthly/NASA-LaRC/CERES-EBAF-4-1/mon/rlut/gn/v20210727/rlut_mon_CERES-EBAF-4-1_PCMDI_gn_200301-201812.nc
outfile: demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.nc
outfilename: rlut_mon_CERES-EBAF-4-1_BE_gn.nc
outpath: demo_output_tmp/climo/
ver: v20241023
infilename: rlut_mon_CERES-EBAF-4-1_PCMDI_gn_200301-201812.nc
type(d): <class 'xarray.core.dataset.Dataset'>
atts: {'Conventions': 'CF-1.7 ODS-2.1', 'activity_id': 'obs4MIPs', 'contact': 'PCMDI (pcmdi-metrics@llnl.gov)', 'creation_date': '2021-07-27T18:20:26Z', 'curation_provenance': 'work-in-progress', 'data_specs_version': '2.1.0', 'external_variables': 'areacella', 'frequency': 'mon', 'further_info_url': 'https://furtherinfo.es-doc.org/CMIP6.NASA-LaRC.CERES-EBAF-4-1.experiment_idsub_experiment_idPCMDI', 'grid': '1x1 degree latitude x longitude', 'grid_label': 'gn', 'history': '2021-07-27T18:20:26Z; CMOR rewrote data to be consistent with obs4MIPs, and CF-1.7 ODS-2.1 standards', 'institution': 'NASA-LaRC (Langley Research Center) Hampton, Va', 'institution_id': 'NASA-LaRC', 'mip_era': 'CMIP6', 'nominal_resolution': '100 km', 'product': 'observations', 'realm': 'atmos', 'release_year': '2019', 'source': 'CERES EBAF (Energy Balanced and Filled) TOA Fluxes. Monthly Averages', 'source_description': 'CERES EBAF (Energy Balanced and Filled) TOA Fluxes. Monthly Averages', 'source_id': 'CERES-EBAF-4-1', 'source_label': 'CERES-EBAF-4-1', 'source_name': 'CERES-EBAF4-1', 'source_type': 'satellite_blended', 'source_version_number': '4.1', 'table_id': 'obs4MIPs_Amon', 'table_info': 'Creation Date:(18 November 2020) MD5:d8a4a72de798e86a999881bdaeb1809e', 'title': 'CERES prepared for obs4MIPs (ODS-v2.1.0)', 'tracking_id': 'hdl:21.14102/d1afa116-4801-4801-b302-43c49b5bb7dc', 'variable_id': 'rlut', 'variant_info': 'Best Estimate', 'variant_label': 'PCMDI', 'license': 'Data in this file processed for obs4MIPs by PCMDI and is for research purposes only.', 'cmor_version': '3.6.1'}
outdir: demo_output_tmp/climo/
start_yr_str is  2003
start_mo_str is  01
end_yr_str is  2018
end_mo_str is  12
[/Users/lee1043/mambaforge/envs/pmp_devel_20240425/lib/python3.10/site-packages/xarray/core/indexing.py:1593](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/mambaforge/envs/pmp_devel_20240425/lib/python3.10/site-packages/xarray/core/indexing.py:1593): PerformanceWarning: Slicing with an out-of-order index is generating 16 times more chunks
  return self.array[key]
[/Users/lee1043/mambaforge/envs/pmp_devel_20240425/lib/python3.10/site-packages/xarray/core/indexing.py:1593](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/mambaforge/envs/pmp_devel_20240425/lib/python3.10/site-packages/xarray/core/indexing.py:1593): PerformanceWarning: Slicing with an out-of-order index is generating 16 times more chunks
  return self.array[key]
output file is demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.200301-201812.AC.v20241023.nc
output file is demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.200301-201812.DJF.v20241023.nc
output file is demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.200301-201812.MAM.v20241023.nc
output file is demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.200301-201812.JJA.v20241023.nc
output file is demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.200301-201812.SON.v20241023.nc
start and end are  2003-01   2018-12
variable list:  ['rlut']
ver: v20241023
[/Users/lee1043/mambaforge/envs/pmp_devel_20240425/lib/python3.10/site-packages/pcmdi_metrics/io/string_constructor.py:43](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/mambaforge/envs/pmp_devel_20240425/lib/python3.10/site-packages/pcmdi_metrics/io/string_constructor.py:43): UserWarning: Keyword 'variable' not provided for filling the template.
  warnings.warn(f"Keyword '{k}' not provided for filling the template.")
var: rlut
infile: demo_data_tmp/obs4MIPs_PCMDI_monthly/NASA-LaRC/CERES-EBAF-4-1/mon/rlut/gn/v20210727/rlut_mon_CERES-EBAF-4-1_PCMDI_gn_200301-201812.nc
outfile: demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.nc
outfilename: None
outpath: None
ver: v20241023
infilename: rlut_mon_CERES-EBAF-4-1_PCMDI_gn_200301-201812.nc
type(d): <class 'xarray.core.dataset.Dataset'>
atts: {'Conventions': 'CF-1.7 ODS-2.1', 'activity_id': 'obs4MIPs', 'contact': 'PCMDI (pcmdi-metrics@llnl.gov)', 'creation_date': '2021-07-27T18:20:26Z', 'curation_provenance': 'work-in-progress', 'data_specs_version': '2.1.0', 'external_variables': 'areacella', 'frequency': 'mon', 'further_info_url': 'https://furtherinfo.es-doc.org/CMIP6.NASA-LaRC.CERES-EBAF-4-1.experiment_idsub_experiment_idPCMDI', 'grid': '1x1 degree latitude x longitude', 'grid_label': 'gn', 'history': '2021-07-27T18:20:26Z; CMOR rewrote data to be consistent with obs4MIPs, and CF-1.7 ODS-2.1 standards', 'institution': 'NASA-LaRC (Langley Research Center) Hampton, Va', 'institution_id': 'NASA-LaRC', 'mip_era': 'CMIP6', 'nominal_resolution': '100 km', 'product': 'observations', 'realm': 'atmos', 'release_year': '2019', 'source': 'CERES EBAF (Energy Balanced and Filled) TOA Fluxes. Monthly Averages', 'source_description': 'CERES EBAF (Energy Balanced and Filled) TOA Fluxes. Monthly Averages', 'source_id': 'CERES-EBAF-4-1', 'source_label': 'CERES-EBAF-4-1', 'source_name': 'CERES-EBAF4-1', 'source_type': 'satellite_blended', 'source_version_number': '4.1', 'table_id': 'obs4MIPs_Amon', 'table_info': 'Creation Date:(18 November 2020) MD5:d8a4a72de798e86a999881bdaeb1809e', 'title': 'CERES prepared for obs4MIPs (ODS-v2.1.0)', 'tracking_id': 'hdl:21.14102/d1afa116-4801-4801-b302-43c49b5bb7dc', 'variable_id': 'rlut', 'variant_info': 'Best Estimate', 'variant_label': 'PCMDI', 'license': 'Data in this file processed for obs4MIPs by PCMDI and is for research purposes only.', 'cmor_version': '3.6.1'}
outdir: demo_output_tmp/climo
start_yr_str is  2003
start_mo_str is  01
end_yr_str is  2018
end_mo_str is  12
[/Users/lee1043/mambaforge/envs/pmp_devel_20240425/lib/python3.10/site-packages/xarray/core/indexing.py:1593](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/mambaforge/envs/pmp_devel_20240425/lib/python3.10/site-packages/xarray/core/indexing.py:1593): PerformanceWarning: Slicing with an out-of-order index is generating 16 times more chunks
  return self.array[key]
[/Users/lee1043/mambaforge/envs/pmp_devel_20240425/lib/python3.10/site-packages/xarray/core/indexing.py:1593](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/mambaforge/envs/pmp_devel_20240425/lib/python3.10/site-packages/xarray/core/indexing.py:1593): PerformanceWarning: Slicing with an out-of-order index is generating 16 times more chunks
  return self.array[key]
output file is demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.200301-201812.AC.v20241023.nc
output file is demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.200301-201812.DJF.v20241023.nc
output file is demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.200301-201812.MAM.v20241023.nc
output file is demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.200301-201812.JJA.v20241023.nc
output file is demo_output_tmp/climo/rlut_mon_CERES-EBAF-4-1_BE_gn.200301-201812.SON.v20241023.nc
Fontconfig warning: ignoring UTF-8: not a valid region tag
2024-11-06 14:51:21,277 [WARNING]: font_manager.py(<lambda>:1040) >> Matplotlib is building the font cache; this may take a moment.
2024-11-06 14:51:21,277 [WARNING]: font_manager.py(<lambda>:1040) >> Matplotlib is building the font cache; this may take a moment.
2024-11-06 14:51:47,580 [INFO]: font_manager.py(__init__:1052) >> Failed to extract font properties from [/System/Library/Fonts/LastResort.otf](https://file+.vscode-resource.vscode-cdn.net/System/Library/Fonts/LastResort.otf): tuple indices must be integers or slices, not str
2024-11-06 14:51:47,580 [INFO]: font_manager.py(__init__:1052) >> Failed to extract font properties from [/System/Library/Fonts/LastResort.otf](https://file+.vscode-resource.vscode-cdn.net/System/Library/Fonts/LastResort.otf): tuple indices must be integers or slices, not str
2024-11-06 14:51:48,028 [INFO]: font_manager.py(__init__:1052) >> Failed to extract font properties from [/System/Library/PrivateFrameworks/FontServices.framework/Resources/Reserved/PingFangUI.ttc](https://file+.vscode-resource.vscode-cdn.net/System/Library/PrivateFrameworks/FontServices.framework/Resources/Reserved/PingFangUI.ttc): In FT2Font: Can not load face (locations (loca) table missing; error code 0x90)
2024-11-06 14:51:48,028 [INFO]: font_manager.py(__init__:1052) >> Failed to extract font properties from [/System/Library/PrivateFrameworks/FontServices.framework/Resources/Reserved/PingFangUI.ttc](https://file+.vscode-resource.vscode-cdn.net/System/Library/PrivateFrameworks/FontServices.framework/Resources/Reserved/PingFangUI.ttc): In FT2Font: Can not load face (locations (loca) table missing; error code 0x90)
2024-11-06 14:51:48,095 [INFO]: font_manager.py(__init__:1052) >> Failed to extract font properties from [/System/Library/Fonts/Supplemental/NISC18030.ttf](https://file+.vscode-resource.vscode-cdn.net/System/Library/Fonts/Supplemental/NISC18030.ttf): In FT2Font: Could not set the fontsize (invalid pixel size; error code 0x17)
2024-11-06 14:51:48,095 [INFO]: font_manager.py(__init__:1052) >> Failed to extract font properties from [/System/Library/Fonts/Supplemental/NISC18030.ttf](https://file+.vscode-resource.vscode-cdn.net/System/Library/Fonts/Supplemental/NISC18030.ttf): In FT2Font: Could not set the fontsize (invalid pixel size; error code 0x17)
2024-11-06 14:51:48,221 [INFO]: font_manager.py(__init__:1052) >> Failed to extract font properties from [/System/Library/Fonts/Apple](https://file+.vscode-resource.vscode-cdn.net/System/Library/Fonts/Apple) Color Emoji.ttc: In FT2Font: Could not set the fontsize (invalid pixel size; error code 0x17)
2024-11-06 14:51:48,221 [INFO]: font_manager.py(__init__:1052) >> Failed to extract font properties from [/System/Library/Fonts/Apple](https://file+.vscode-resource.vscode-cdn.net/System/Library/Fonts/Apple) Color Emoji.ttc: In FT2Font: Could not set the fontsize (invalid pixel size; error code 0x17)
2024-11-06 14:51:48,270 [INFO]: font_manager.py(_load_fontmanager:1584) >> generated new fontManager
2024-11-06 14:51:48,270 [INFO]: font_manager.py(_load_fontmanager:1584) >> generated new fontManager
Traceback (most recent call last):
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/bin/pcmdi_compute_climatologies.py", line 6, in <module>
    from pcmdi_metrics.mean_climate.lib import calculate_climatology
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/pcmdi_metrics/mean_climate/lib/__init__.py", line 21, in <module>
    from .create_mean_climate_parser import create_mean_climate_parser  # noqa
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/pcmdi_metrics/mean_climate/lib/create_mean_climate_parser.py", line 4, in <module>
    from pcmdi_metrics.mean_climate.lib import pmp_parser
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/pcmdi_metrics/mean_climate/lib/pmp_parser.py", line 6, in <module>
    from pcmdi_metrics.mean_climate.lib.pmp_parameter import (
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/pcmdi_metrics/mean_climate/lib/pmp_parameter.py", line 7, in <module>
    from pcmdi_metrics.utils import StringConstructor
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/pcmdi_metrics/utils/__init__.py", line 20, in <module>
    from .land_sea_mask import apply_landmask, apply_oceanmask, create_land_sea_mask
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/pcmdi_metrics/utils/land_sea_mask.py", line 9, in <module>
    import regionmask
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/regionmask/__init__.py", line 7, in <module>
    from . import core, defined_regions
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/regionmask/defined_regions/__init__.py", line 3, in <module>
    from . import _natural_earth, _ressources
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/regionmask/defined_regions/_natural_earth.py", line 12, in <module>
    from ..core.regions import Regions
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/regionmask/core/regions.py", line 15, in <module>
    from .mask import _inject_mask_docstring, _mask_2D, _mask_3D
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/regionmask/core/mask.py", line 420, in <module>
    mask, lon, lat, polygons, numbers, fill=np.NaN, is_unstructured=False
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.NaN` was removed in the NumPy 2.0 release. Use `np.nan` instead.. Did you mean: 'nan'?

lee1043 · 2024-11-06T23:19:11Z

(edited) Check boxes for demo notebook moved to the top of this PR.

lee1043 · 2024-11-06T23:34:38Z

Error from MoV

%%bash
variability_modes_driver.py -p basic_mov_param.py --case_id "mov_1"

mip: cmip5
exp: historical
fq: mo
realm: atm
EofScaling: False
RmDomainMean: True
LandMask: False
nc_out_obs, plot_obs: True True
nc_out_model, plot_model: True True
CMEC:False
mode: NAM
seasons: ['DJF']
models: ['ACCESS1-0']
number of models: 1
realization:  r1i1p1
eofn_obs: 1
eofn_mod: 1
parallel: False
output directories:
graphics : demo_output_tmp/mov_1
diagnostic_results : demo_output_tmp/mov_1
metrics_results : demo_output_tmp/mov_1
Converting units by  divide 100.0
Traceback (most recent call last):
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/bin/variability_modes_driver.py", line 411, in <module>
    ) = eof_analysis_get_variance_mode(
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/pcmdi_metrics/variability_mode/lib/eof_analysis.py", line 64, in eof_analysis_get_variance_mode
    solver = Eof(da, weights=area_weights)
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/eofs/xarray.py", line 130, in __init__
    self._solver = standard.Eof(array.data,
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/eofs/standard.py", line 124, in __init__
    channels = np.product(self._originalshape)
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/numpy/__init__.py", line 410, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'product'

Related issue: ajdawson/eofs#151
PR for the above issue: ajdawson/eofs#148

lee1043 · 2024-11-07T00:50:03Z

Error from MJO

Click here for Error message

%%bash
mjo_metrics_driver.py -p basic_mjo_param.py

includeOBS: True
models: ['GISS-E2-H']
realization:  r6i1p1
output directories:
graphics : demo_output_tmp/mjo/Ex1
diagnostic_results : demo_output_tmp/mjo/Ex1
metrics_results : demo_output_tmp/mjo/Ex1
CMEC: False
debug:  False
parallel: False
 -----  obs  ---------------------
 ---  GPCP-IP  ---
demo_data_tmp/obs4MIPs_PCMDI_daily/NASA-JPL/GPCP-1-3/day/pr/gn/latest/pr_day_GPCP-1-3_PCMDI_gn_19961002-20170101.nc
 --  NDJFMA  --
1998
Converting units by  multiply 86400.0
warning: failed for  obs GPCP-IP module 'numpy' has no attribute 'float'.
`np.float` was a deprecated alias for the builtin `float`. To avoid this error in existing code, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
 -----  GISS-E2-H  ---------------------
 ---  r6i1p1  ---
demo_data_tmp/CMIP5_demo_timeseries/historical/atmos/day/pr/pr_day_GISS-E2-H_historical_r6i1p1_20000101-20051231.nc
 --  NDJFMA  --
2000
Converting units by  multiply 86400.0
2001
Converting units by  multiply 86400.0
warning: failed for  GISS-E2-H r6i1p1 module 'numpy' has no attribute 'float'.
`np.float` was a deprecated alias for the builtin `float`. To avoid this error in existing code, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

lee1043 · 2024-11-07T00:53:01Z

Error from ENSO:

%%bash
enso_driver.py --help

Traceback (most recent call last):
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/bin/enso_driver.py", line 16, in <module>
    from EnsoMetrics.EnsoCollectionsLib import (
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/EnsoMetrics/__init__.py", line 2, in <module>
    from .EnsoComputeMetricsLib import *
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/EnsoMetrics/EnsoComputeMetricsLib.py", line 11, in <module>
    from .EnsoMetricsLib import BiasPrLatRmse, BiasPrLonRmse, BiasPrRmse, BiasSshLatRmse, BiasSshLonRmse, BiasSshRmse,\
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/EnsoMetrics/EnsoMetricsLib.py", line 12, in <module>
    from .EnsoUvcdatToolsLib import ArrayListAx, ArrayToList, AverageMeridional, AverageZonal, BasinMask, CheckTime,\
  File "/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/EnsoMetrics/EnsoUvcdatToolsLib.py", line 14, in <module>
    from numpy import product as NPproduct
ImportError: cannot import name 'product' from 'numpy' (/Users/lee1043/mambaforge/envs/pmp_devel_20241106_xcdat0.7.3/lib/python3.10/site-packages/numpy/__init__.py)

lee1043 · 2024-11-07T01:22:34Z

Error from Demo 7 precip variavbilitiy

%%bash -s "$demo_output_directory"
variability_across_timescales_PS_driver.py -p basic_precip_variability_param.py \
--region_file $1/shp/CONUS.shp \
--attr 'Region' \
--feature 'CONUS' \
--results_dir $1/precip_variability/region_ex

Error message:

Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?c9ec7769-567c-4614-ae79-06a086fe1c01) or open in a [text editor](command:workbench.action.openLargeOutput?c9ec7769-567c-4614-ae79-06a086fe1c01). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...
demo_data_tmp/CMIP5_demo_timeseries/historical/atmos/day/pr/
pr_day_GISS-E2-H_historical_r6i1p1_*.nc
[2000, 2005]
730 365
2
demo_output/precip_variability/region_ex
demo_output/precip_variability/region_ex
demo_output/precip_variability/region_ex
['demo_data_tmp/CMIP5_demo_timeseries/historical/atmos/day/pr/pr_day_GISS-E2-H_historical_r6i1p1_20000101-20051231.nc']
GISS-E2-H.r6i1p1
['demo_data_tmp/CMIP5_demo_timeseries/historical/atmos/day/pr/pr_day_GISS-E2-H_historical_r6i1p1_20000101-20051231.nc']
GISS-E2-H.r6i1p1 365_day
2000 2005
Complete regridding from (2190, 90, 144) to (2190, 90, 180)
Cropping from shapefile
Reading region from file.
bash: line 5: 84435 Segmentation fault: 11  variability_across_timescales_PS_driver.py -p basic_precip_variability_param.py --region_file $1/shp/CONUS.shp --attr 'Region' --feature 'CONUS' --results_dir $1/precip_variability/region_ex

lee1043 · 2024-11-07T01:53:46Z

Error from Demo 8:

%%bash
extremes_driver.py  -p basic_extremes_param.py

This does not generate the output directory extremes_1

lee1043 · 2024-11-07T01:58:08Z

Not surprisingly, bunch of errors raised from demo notebooks, likely resulted from the numpy version update. Demo notebooks 0 to 3 run okay, but others are failing.

@acordonez I will work on notebooks 4 to 6, and can I get you help on notebooks 7 to 9? Just a reminder, to test these, it is needed to generate a new conda env using the dev.yml in this branch.

lee1043 · 2024-11-07T20:15:19Z

Regarding the error with ENSO, a PR. made for the ENSO package: CLIVAR-PRP/ENSO_metrics#80

lee1043 · 2024-11-19T00:43:52Z

Confirmed now demo 6 ENSO notebook working without issues after updating enso_metrics version to 1.1.2.

lee1043 · 2024-11-19T00:45:40Z

Confirmed MoV notebook also working after updating eofs to 2.0.0.

Update build_workflow.yml

acordonez · 2024-11-20T23:41:02Z

@lee1043 I didn't find any problems with notebooks 8, 9, or 9b. In the comment above I documented some of the experimentation I did with the precip variability metrics and have one suggestion for a workaround.

lee1043 · 2024-11-20T23:48:09Z

@acordonez thanks for the update. Great to hear there were no issue with 8, 9, and 9b. What do you consider as the most efficient and quick solution for the issue with 7?

By the way, I found that CI build test was failing with the following message. Have you seen something like this? I am yet to have good idea how to address it.

Updating 'pcmdi_metrics_ci' env from conda env update...
  /home/runner/miniconda3/condabin/mamba env create --name pcmdi_metrics_ci --file conda-env/ci.yml
  Warning: /home/runner/miniconda3/lib/python3.12/argparse.py:2006: FutureWarning: `remote_definition` is deprecated and will be removed in 25.9. Use `conda env create --file=URL` instead.
    action(self, namespace, argument_values, option_string)
  
  /home/runner/miniconda3/lib/python3.12/argparse.py:2006: FutureWarning: `remote_definition` is deprecated and will be removed in 25.9. Use `conda env create --file=URL` instead.
    action(self, namespace, argument_values, option_string)
  Channels:
   - conda-forge
   - defaults
  Platform: linux-64
  Collecting package metadata (repodata.json): ...working... done
  Solving environment: ...working... ::warning::warning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0Awarning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY%0A
  warning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY

acordonez · 2024-11-21T00:23:37Z

@lee1043 For the precip variability metrics, this is the solution I found that worked in all cases:
Change Line 86 to rgtmp = RegridHoriz(f.compute(), var, res, regions_specs, lon_range).

I'm not sure about the conda build failure. It seems like it might be having difficulty solving the environment and either failing or timing out as a result?

lee1043 · 2024-11-21T00:35:02Z

@acordonez thank you for the suggestion. I will make that change to this PR.

Regarding the conda build, I think the above changes in the yml files helped passing the test. In summary, I fixed some versions of dependencies, and set conda channel to be flexible, and set to use conda classic solver.

lee1043 · 2024-11-21T01:22:18Z

Confirmed demo 7 working with no noticeable issue and the results are consistent.

lee1043 · 2024-11-26T22:09:46Z

Merging the PR after reviewed by @lee1043 and @acordonez and all CI tests passed.

update versions of numpy, cdms, xcdat

c0f5edc

lee1043 added this to the 3.7 milestone Oct 29, 2024

lee1043 self-assigned this Oct 29, 2024

lee1043 linked an issue Oct 29, 2024 that may be closed by this pull request

[Feature]: Removal of CDAT, cdms, cdutil, etc. #1163

Closed

lee1043 added the build label Oct 29, 2024

lee1043 and others added 2 commits October 29, 2024 15:19

update xcdat version to 0.7.3 to work with numpy >= 2.0

5869235

Merge branch 'main' into feature/1163_numpy-upgrade_lee1043

1112647

lee1043 modified the milestones: 3.7, 3.8 Oct 30, 2024

Merge branch 'main' into feature/1163_numpy-upgrade_lee1043

35bdee5

lee1043 added 2 commits November 6, 2024 14:29

cartopy version conflicts with numpy, so released regulation

3525255

released version regulations for shapely and matplotlib to avoid conf…

1d304b0

…lict with updated numpy version

update regionmask version, which resolved the error in #1164 (comment)

bf0514a

resolves error in #1164 (comment)

4a92562

lee1043 and others added 6 commits November 18, 2024 14:37

Merge branch 'main' into feature/1163_numpy-upgrade_lee1043

fc6475f

update versions for eofs and enso

b4736dd

Merge branch 'main' into feature/1163_numpy-upgrade_lee1043

06d1d66

Merge branch 'main' into feature/1163_numpy-upgrade_lee1043

a395367

Merge branch 'main' into feature/1163_numpy-upgrade_lee1043

54ed820

Merge branch 'main' into feature/1163_numpy-upgrade_lee1043

b72c0a0

lee1043 added CRITICAL QC Quality Control labels Nov 19, 2024

lee1043 added 2 commits November 18, 2024 20:23

Update build_workflow.yml

ff13344

Merge pull request #1197 from PCMDI/lee1043-patch-1

734deb6

Update build_workflow.yml

Update build_workflow.yml

e707e4b

lee1043 marked this pull request as ready for review November 21, 2024 00:07

lee1043 marked this pull request as draft November 21, 2024 00:07

lee1043 added 4 commits November 20, 2024 16:19

Update dev.yml

592f64b

Update ci.yml

77faf8d

Update dev.yml

ed91962

Update ci.yml

b2b3830

change following #1164 (comment)

25a91bd

lee1043 marked this pull request as ready for review November 21, 2024 16:41

Merge branch 'main' into feature/1163_numpy-upgrade_lee1043

ec022df

lee1043 merged commit aaa8a86 into main Nov 26, 2024
4 checks passed

lee1043 deleted the feature/1163_numpy-upgrade_lee1043 branch November 26, 2024 22:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Numpy version update #1164

Numpy version update #1164

lee1043 commented Oct 29, 2024 •

edited

Loading

lee1043 commented Oct 29, 2024 •

edited

Loading

tomvothecoder commented Oct 29, 2024

lee1043 commented Oct 29, 2024

tomvothecoder commented Oct 31, 2024

lee1043 commented Nov 6, 2024

tomvothecoder commented Nov 6, 2024

lee1043 commented Nov 6, 2024 •

edited

Loading

lee1043 commented Nov 6, 2024 •

edited

Loading

lee1043 commented Nov 6, 2024 •

edited

Loading

lee1043 commented Nov 6, 2024 •

edited

Loading

lee1043 commented Nov 7, 2024 •

edited

Loading

lee1043 commented Nov 7, 2024

lee1043 commented Nov 7, 2024

lee1043 commented Nov 7, 2024

lee1043 commented Nov 7, 2024

lee1043 commented Nov 7, 2024 •

edited

Loading

lee1043 commented Nov 19, 2024

lee1043 commented Nov 19, 2024

acordonez commented Nov 20, 2024

lee1043 commented Nov 20, 2024

acordonez commented Nov 21, 2024

lee1043 commented Nov 21, 2024 •

edited

Loading

lee1043 commented Nov 21, 2024 •

edited

Loading

lee1043 commented Nov 26, 2024

Numpy version update #1164

Numpy version update #1164

Conversation

lee1043 commented Oct 29, 2024 • edited Loading

lee1043 commented Oct 29, 2024 • edited Loading

tomvothecoder commented Oct 29, 2024

lee1043 commented Oct 29, 2024

tomvothecoder commented Oct 31, 2024

lee1043 commented Nov 6, 2024

tomvothecoder commented Nov 6, 2024

lee1043 commented Nov 6, 2024 • edited Loading

lee1043 commented Nov 6, 2024 • edited Loading

lee1043 commented Nov 6, 2024 • edited Loading

lee1043 commented Nov 6, 2024 • edited Loading

lee1043 commented Nov 7, 2024 • edited Loading

lee1043 commented Nov 7, 2024

lee1043 commented Nov 7, 2024

lee1043 commented Nov 7, 2024

lee1043 commented Nov 7, 2024

lee1043 commented Nov 7, 2024 • edited Loading

lee1043 commented Nov 19, 2024

lee1043 commented Nov 19, 2024

acordonez commented Nov 20, 2024

lee1043 commented Nov 20, 2024

acordonez commented Nov 21, 2024

lee1043 commented Nov 21, 2024 • edited Loading

lee1043 commented Nov 21, 2024 • edited Loading

lee1043 commented Nov 26, 2024

lee1043 commented Oct 29, 2024 •

edited

Loading

lee1043 commented Oct 29, 2024 •

edited

Loading

lee1043 commented Nov 6, 2024 •

edited

Loading

lee1043 commented Nov 6, 2024 •

edited

Loading

lee1043 commented Nov 6, 2024 •

edited

Loading

lee1043 commented Nov 6, 2024 •

edited

Loading

lee1043 commented Nov 7, 2024 •

edited

Loading

lee1043 commented Nov 7, 2024 •

edited

Loading

lee1043 commented Nov 21, 2024 •

edited

Loading

lee1043 commented Nov 21, 2024 •

edited

Loading