Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use smaller testdata for Raven tutorial notebook so can enable Raven notebooks on Jenkins #383

Closed
tlvu opened this issue Apr 20, 2021 · 4 comments
Assignees

Comments

@tlvu
Copy link
Contributor

tlvu commented Apr 20, 2021

The goal is to enable Raven tutorial notebooks on Jenkins, just like Finch and Pavics-sdi notebooks currently.

This means Raven notebooks should pass on all test deployments of PAVICS, not just on prod deployment. So we need to replicate testdata to all the test deployments.

This notebook https://github.com/Ouranosinc/raven/blob/3c768f167fbb7515206b63b90020e57708ebbc4d/docs/source/notebooks/Full_process_example_1.ipynb use two gigantic testdata file (ERA5_tas = "https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/ecmwf/era5_rechunk/tas_era5_reanalysis_hourly_2020.nc", ERA5_pr = "https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/ecmwf/era5_rechunk/pr_era5_reanalysis_hourly_2020.nc") that are 17G each !

Either:

  • change for other smaller testdata
  • change for .ncml link instead of .nc (.ncml do not need to be replicated to test deployments, they are taken directly on the prod server)
  • change the hostname from pavics.ouranos.ca to boreas.ouranos.ca so these .nc files are taken directly from prod server instead of from the test server
@matprov
Copy link

matprov commented May 17, 2021

@tlvu Just by curiosity, any update on this one? This would allow us to run the e2e raven tests, which are required for all the PRs regarding raven.

@tlvu
Copy link
Contributor Author

tlvu commented May 17, 2021

@tlvu Just by curiosity, any update on this one? This would allow us to run the e2e raven tests, which are required for all the PRs regarding raven.

@huard ? I have to defer to you as I do not understand the notebook enough to be able to make the call.

@huard
Copy link
Contributor

huard commented May 17, 2021

I made this choice initially because the performance with the NcML was terrible. It just hanged there forever. Since the NcML is now using rechunked date, we should check if performance is acceptable.

@richardarsenault
Copy link
Contributor

Smaller files have been provided and the Full_process notebook has been deprecated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants