Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add horizontal (CPU) scaling via dask #54

Open
tcmetzger opened this issue Oct 11, 2024 · 1 comment
Open

Add horizontal (CPU) scaling via dask #54

tcmetzger opened this issue Oct 11, 2024 · 1 comment
Assignees
Milestone

Comments

@tcmetzger
Copy link
Collaborator

The Python user interface will take advantage of horizontal (CPU) scaling options provided by xarray. Specifically, when a user supplies the chunks argument when loading data in xarray, the xarray.DataArray is backed by a dask.Array which requires consideration by pyRTE-RRTMPG.

This task also includes creating a notebook of a global example (roughly 1 degree resolution at one point in time) using the existing clear_sky problem with a snapshot of the globe (Robert to obtain data).

@tcmetzger tcmetzger added this to the Phase 2 milestone Oct 11, 2024
@tcmetzger
Copy link
Collaborator Author

tcmetzger commented Oct 28, 2024

The wrinkle here is that the underlying kernel expects chunks - how big should they be? we need a reasonable default, depending on the platform or hardware - something that the user can override.

Potentially: chunking in Python, then call multiple instances of Fortran without chunks? However, this might have severe performance drawbacks.

Potentially: chunk in python by default, but also allow user to pass on a param to Fortran.

@tcmetzger tcmetzger moved this from To Do to In progress in pyRTE Phase 2 Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In progress
Development

No branches or pull requests

2 participants