Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

weaver: bump version to 5.1.1 #439

Closed
wants to merge 14 commits into from
Closed

Conversation

mishaschwartz
Copy link
Collaborator

@mishaschwartz mishaschwartz commented Mar 19, 2024

Overview

See changes in the Weaver changes file

Changes

Non-breaking changes

  • New component version Weaver: 5.1.1

Breaking changes

Related Issue / Discussion

Additional Information

CI Operations

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

@github-actions github-actions bot added component/weaver Related to https://github.com/crim-ca/weaver documentation Improvements or additions to documentation feature/WPS Feature or service related to Web Processing Service labels Mar 19, 2024
@mishaschwartz mishaschwartz changed the title weaver: bump version to 5.1.0 weaver: bump version to 5.1.1 Mar 20, 2024
@mishaschwartz mishaschwartz marked this pull request as ready for review March 25, 2024 13:19
@mishaschwartz
Copy link
Collaborator Author

run tests

Copy link
Collaborator

@tlvu tlvu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well that's a simple PR :D Could you maybe explain very briefly in the changelog what fixes or features we are trying to get with the new Weaver? If too long to summarize, maybe just put the Weaver changelog diff URL between 4.32.0 and 5.1.1.

@mishaschwartz
Copy link
Collaborator Author

@fmigneault any idea why the jenkins tests aren't running for this one? It would be useful to double check that none of the weaver tests fail with the update.

@fmigneault
Copy link
Collaborator

run tests

@fmigneault
Copy link
Collaborator

@mishaschwartz
Can't figure out why it is not triggered for this PR. The Jenkins CI does not show any build referring to this PR. One would have to inspect the GitHub hooks if an event was sent, which is what the CI is listening for. I do not have the necessary access to inspect hooks from GitHub.

As a workaround in the meantime, I triggered it manually from Jenkins: http://daccs-jenkins.crim.ca/job/DACCS-iac-birdhouse/2560/

Copy link
Collaborator

@fmigneault fmigneault left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine, but waiting on what the CI responds for test execution...

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2560/
ResultFAILURE

BIRDHOUSE_DEPLOY_BRANCH : bump-weaver-5.1.0
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1562/

NOTEBOOK TEST RESULTS
    
[2024-03-25T17:52:10.683Z] ============================= test session starts ==============================
[2024-03-25T17:52:10.683Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2024-03-25T17:52:10.683Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2024-03-25T17:52:10.683Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2024-03-25T17:52:10.683Z] collected 270 items
[2024-03-25T17:52:10.683Z] 
[2024-03-25T17:52:23.306Z] notebooks-auth/geoserver.ipynb ..................                        [  6%]
[2024-03-25T17:53:02.789Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [ 10%]
[2024-03-25T17:53:08.116Z] notebooks-auth/test_thredds.ipynb ...........                            [ 14%]
[2024-03-25T17:53:59.728Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [ 16%]
[2024-03-25T17:54:08.309Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 19%]
[2024-03-25T17:54:18.290Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 21%]
[2024-03-25T17:54:28.341Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 24%]
[2024-03-25T18:01:51.781Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 28%]
[2024-03-25T18:01:51.781Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 29%]
[2024-03-25T18:01:58.506Z] ...............                                                          [ 34%]
[2024-03-25T18:02:07.203Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 36%]
[2024-03-25T18:02:15.840Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 38%]
[2024-03-25T18:02:31.630Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 41%]
[2024-03-25T18:02:43.190Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 43%]
[2024-03-25T18:02:47.432Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 45%]
[2024-03-25T18:06:28.369Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 51%]
[2024-03-25T18:07:42.294Z] .............                                                            [ 55%]
[2024-03-25T18:07:44.469Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 57%]
[2024-03-25T18:07:46.427Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 60%]
[2024-03-25T18:08:03.176Z] .................                                                        [ 66%]
[2024-03-25T18:08:10.611Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 68%]
[2024-03-25T18:08:11.996Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 69%]
[2024-03-25T18:08:28.996Z] ........F                                                                [ 72%]
[2024-03-25T18:08:39.036Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 76%]
[2024-03-25T18:08:48.047Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 78%]
[2024-03-25T18:08:49.429Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2024-03-25T18:08:52.996Z] ......                                                                   [ 81%]
[2024-03-25T18:09:01.126Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 81%]
[2024-03-25T18:09:16.488Z] .............                                                            [ 86%]
[2024-03-25T18:09:26.516Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2024-03-25T18:10:11.143Z] ....s.                                                                   [ 89%]
[2024-03-25T18:10:17.739Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2024-03-25T18:10:33.277Z] ...                                                                      [ 90%]
[2024-03-25T18:10:45.504Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2024-03-25T18:11:07.818Z] ......                                                                   [ 93%]
[2024-03-25T18:11:10.551Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2024-03-25T18:13:44.928Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2024-03-25T18:13:44.928Z] 
[2024-03-25T18:13:44.928Z] =================================== FAILURES ===================================
    
  

@fmigneault
Copy link
Collaborator

@mishaschwartz
Only the final cell of https://github.com/Ouranosinc/pavics-sdi/blob/master/docs/source/notebook-components/weaver_example.ipynb is failing.
The cell checks that the status is succeeded from Weaver's point of view, so I assume the underlying Hummingbird ncdump process dumped a plain Error: generating ncdump failed. Output: [... actual NetCDF contents ...] in the text file, or that more stderr/stdout get captured somehow and piped to that file. Needs more investigation, but I don't have anything obvious in mind that can pinpoint the cause.

14:13:44   mismatch 'stdout'
14:13:44  
14:13:44   assert reference_output == test_output failed:
14:13:44  
14:13:44    '\nJob was su...e" ;\n}\n\n\n' == '\nJob was su...e" ;\n}\n\n\n'
14:13:44    Skipping 194 identical leading characters in diff, use -v to show
14:13:44      content:
14:13:44      
14:13:44    - Error: generating ncdump failed. Output: netcdf ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc {
14:13:44    ? -----------------------------------------
14:13:44    + netcdf ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc {
14:13:44      dimensions:
14:13:44      	time = UNLIMITED ; // (120 currently)
14:13:44      	bnds = 2 ;
14:13:44      	lat = 160 ;
14:13:44      	lon = 320 ;
14:13:44      	plev = 23 ;
14:13:44      variables:
14:13:44      	double time(time) ;
14:13:44      		time:bounds = "time_bnds" ;
14:13:44      		time:units = "days since 1981-01-01" ;
14:13:44      		time:calendar = "standard" ;
14:13:44      		time:axis = "T" ;
14:13:44      		time:long_name = "time" ;
14:13:44      		time:standard_name = "time" ;
14:13:44      	double time_bnds(time, bnds) ;
14:13:44      	double plev(plev) ;
14:13:44      		plev:units = "Pa" ;
14:13:44      		plev:axis = "Z" ;
14:13:44      		plev:positive = "down" ;
14:13:44      		plev:long_name = "pressure" ;
14:13:44      		plev:standard_name = "air_pressure" ;
14:13:44      	double lat(lat) ;
14:13:44      		lat:bounds = "lat_bnds" ;
14:13:44      		lat:units = "degrees_north" ;
14:13:44      		lat:axis = "Y" ;
14:13:44      		lat:long_name = "latitude" ;
14:13:44      		lat:standard_name = "latitude" ;
14:13:44      	double lat_bnds(lat, bnds) ;
14:13:44      	double lon(lon) ;
14:13:44      		lon:bounds = "lon_bnds" ;
14:13:44      		lon:units = "degrees_east" ;
14:13:44      		lon:axis = "X" ;
14:13:44      		lon:long_name = "longitude" ;
14:13:44      		lon:standard_name = "longitude" ;
14:13:44      	double lon_bnds(lon, bnds) ;
14:13:44      	float ta(time, plev, lat, lon) ;
14:13:44      		ta:standard_name = "air_temperature" ;
14:13:44      		ta:long_name = "Air Temperature" ;
14:13:44      		ta:units = "K" ;
14:13:44      		ta:original_name = "T" ;
14:13:44      		ta:cell_methods = "time: mean (interval: 30 minutes)" ;
14:13:44      		ta:cell_measures = "area: areacella" ;
14:13:44      		ta:history = "DATE_TIME_TZ altered by CMOR: replaced missing value flag (-9.99e+33) with standard missing value (1e+20)." ;
14:13:44      		ta:missing_value = 1.e+20f ;
14:13:44      		ta:_FillValue = 1.e+20f ;
14:13:44      		ta:associated_files = "baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_MRI-CGCM3_decadal1980_r0i0p0.nc areacella: areacella_fx_MRI-CGCM3_decadal1980_r0i0p0.nc" ;
14:13:44      
14:13:44      // global attributes:
14:13:44      		:institution = "MRI (Meteorological Research Institute, Tsukuba, Japan)" ;
14:13:44      		:institute_id = "MRI" ;
14:13:44      		:experiment_id = "decadal1980" ;
14:13:44      		:source = "MRI-CGCM3 2011 atmosphere: GSMUV (gsmuv-110112, TL159L48); ocean: MRI.COM3 (MRICOM-3_0-20101116, 1x0.5L51); sea ice: MRI.COM3; land: HAL (HAL_cmip5_v0.31_04); aerosol: MASINGAR-mk2 (masingar_mk2-20110111_0203, TL95L48)" ;
14:13:44      		:model_id = "MRI-CGCM3" ;
14:13:44      		:forcing = "GHG, SA, Oz, LU, Sl, Vl, BC, OC (GHG includes CO2, CH4, N2O, CFC-11, CFC-12, and HCFC-22)" ;
14:13:44      		:parent_experiment_id = "N/A" ;
14:13:44      		:parent_experiment_rip = "N/A" ;
14:13:44      		:branch_time = 0. ;
14:13:44      		:contact = "Seiji Yukimoto (yukimoto@mri-jma.go.jp)" ;
14:13:44      		:history = "Output from /sharex3/cmip5/decadal1980/run-C3_decadal1980_01a/grads/atm_avr_mon.ctl DATE_TIME_TZ CMOR rewrote data to comply with CF standards and CMIP5 requirements." ;
14:13:44      		:references = "Model described by Yukimoto et al. (Technical Report of the Meteorological Research Institute, 2011, 64, 83pp.)" ;
14:13:44      		:initialization_method = 1 ;
14:13:44      		:physics_version = 1 ;
14:13:44      		:tracking_id = "ce91e727-5f22-44fc-b24d-5bb53393ac69" ;
14:13:44      		:product = "output" ;
14:13:44      		:experiment = "10- or 30-year run initialized in year 1980" ;
14:13:44      		:frequency = "mon" ;
14:13:44      		:creation_date = "DATE_TIME_TZ" ;
14:13:44      		:Conventions = "CF-1.4" ;
14:13:44      		:project_id = "CMIP5" ;
14:13:44      		:table_id = "Table Amon (26 July 2011) 976b7fd1d9e1be31dddd28f5dc79b7a1" ;
14:13:44      		:title = "MRI-CGCM3 model output prepared for CMIP5 10- or 30-year run initialized in year 1980" ;
14:13:44      		:parent_experiment = "N/A" ;
14:13:44      		:modeling_realm = "atmos" ;
14:13:44      		:realization = 1 ;
14:13:44      		:cmor_version = "2.7.1" ;
14:13:44      		:DODS_EXTRA.Unlimited_Dimension = "time" ;
14:13:44      }
14:13:44   

@mishaschwartz
Copy link
Collaborator Author

@fmigneault

Needs more investigation, but I don't have anything obvious in mind that can pinpoint the cause.

Ok thanks! I've been investigating on my side but I haven't been able to get a consistent result to debug. If you have some time in the near future could you please have a look as well?

@mishaschwartz
Copy link
Collaborator Author

run tests

@crim-jenkins-bot
Copy link
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2660/
ResultFAILURE

BIRDHOUSE_DEPLOY_BRANCH : bump-weaver-5.1.0
DACCS_IAC_BRANCH :
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

⚠️ Infrastructure deployment failed. ⚠️
Instance destroyed due to CI execution.
To debug, launch an instance manually with PR reference
bump-weaver-5.1.0.

@mishaschwartz
Copy link
Collaborator Author

@fmigneault

I looks like the tests may be failing because the test data was never actually downloaded due to a permission denied error.
If you look at the output logs from the jenkins setup (https://daccs-jenkins.crim.ca/job/DACCS-iac-birdhouse/2560/console):

+ [ ! -f testdata/flyingpigeon/cmip3/pr.sresa2.miub_echo_g.run1.atm.da.nc ]
+ dirname testdata/flyingpigeon/cmip3/pr.sresa2.miub_echo_g.run1.atm.da.nc
+ PARENT_DIRS=testdata/flyingpigeon/cmip3
+ [ ! -d testdata/flyingpigeon/cmip3 ]
+ mkdir -p testdata/flyingpigeon/cmip3
mkdir: cannot create directory ���testdata���: Permission denied
+ curl https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/fileServer/birdhouse/testdata/flyingpigeon/cmip3/pr.sresa2.miub_echo_g.run1.atm.da.nc --output testdata/flyingpigeon/cmip3/pr.sresa2.miub_echo_g.run1.atm.da.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to create the file 
Warning: testdata/flyingpigeon/cmip3/pr.sresa2.miub_echo_g.run1.atm.da.nc: No 
Warning: such file or directory

  0  570k    0  3696    0     0  28253      0  0:00:20 --:--:--  0:00:20 28430
curl: (23) Failure writing output to destination

It seems that the user that runs this stage in the build does not have permission to create directories at the correct location so there is no directory to put the file when it is downloaded with curl.

This is just one example, if you look at the log output you can see that this same error occurs for every file downloaded through the bootstrap-testdata script.

Do you mind checking this out?

@fmigneault
Copy link
Collaborator

fmigneault commented May 23, 2024

Doesn't look like there were any changes related to test data setup since #439 (comment), but that one ran. Not sure of the cause, but will wait to see if it can be replicated.

@fmigneault
Copy link
Collaborator

fmigneault commented May 23, 2024

@mishaschwartz
Copy link
Collaborator Author

Is it possible that running the tests with the "run tests" comment runs with a different user than if the test is triggered by GitHub?

@fmigneault
Copy link
Collaborator

Is it possible that running the tests with the "run tests" comment runs with a different user than if the test is triggered by GitHub?

Yes. There is a specific user for CI triggers. However, it is not that user that is used within the instance doing the operations. This should be the "same" ubuntu user created dynamically for respective instances.

fmigneault added a commit that referenced this pull request Jun 20, 2024
## Overview

See changes in https://pavics-weaver.readthedocs.io/en/latest/changes.html#changes-5-6-1
Running here: https://hirondelle.crim.ca/weaver/versions
Corresponding changes evaluated during OGC initiatives with cross-platform provider interoperability tests.

## Changes

**Non-breaking changes**

- Weaver: bump version to [5.6.1](https://github.com/crim-ca/weaver/tree/5.6.1).

  - See full changes details in [Weaver changes](https://pavics-weaver.readthedocs.io/en/latest/changes.html#changes-5-6-1)
  - In summary:
    - multiple control setting options to customize some behaviors
    - improved *OGC API - Processes* standard conformance
    - improved support of *Common Workflow Language (CWL)* features (secrets, sub-workflow, auth-propagation, etc.)

- Weaver: WPS retry logic on post-compose step.
  - Apply `--network birdhouse_default` to the Docker `curl` image to allow HTTP requests to properly resolve
    against the running services (WPS bird providers, Weave and Magpie). In some cases, this network would not
    be automatically resolved.
  - Fix the index used during HTTP request retry to avoid going one step over the intended retry attempts.

**Breaking changes**
- n/a

## Related Issue / Discussion

- Closes #439
  Merge the PR since it has issues running the CI somehow. Replace by this one all together. 

## Additional Information

## CI Operations

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/weaver Related to https://github.com/crim-ca/weaver documentation Improvements or additions to documentation feature/WPS Feature or service related to Web Processing Service
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants