Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply Dask to parallelise acs_regional_stats #34

Open
xenct opened this issue Dec 17, 2024 · 0 comments
Open

Apply Dask to parallelise acs_regional_stats #34

xenct opened this issue Dec 17, 2024 · 0 comments

Comments

@xenct
Copy link
Collaborator

xenct commented Dec 17, 2024

acs_regional_stats can be very memory intensive to run, particularly over many regions and many timesteps.
We should develop an example of running acs_regional_stats for many years of daily data to produce area averaged timeseries for regions. Currently, this is possible, but will take several minutes to calculate.
Dask is likely to be able to achieve this by calculating area averages per file.
Previous development has focused on reducing memory usage through other clever means, such as implementing chunks to reduce the number of timesteps loaded into the memory to calculate stats over each time. This could be parallelised, but it is not currently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant