Load hazard raster data lazily

[peanutfun]

Changes proposed in this PR:

• Use dask array chunks to load hazard raster data lazily
• Add sparse as new dependency for that task

This PR fixes #544, an issue raised during the review of #507

PR Author Checklist

• Read the Contribution Guide
• Correct target branch selected (if unsure, select develop)
• Source branch up-to-date with target branch
• Documentation updated
• Tests updated
• Tests passing
• No new linter issues

PR Reviewer Checklist

• Read the Contribution Guide
• CLIMADA Reviewer Checklist passed
• Tests passing
• No new linter issues

[timschmi95]

This change does enable to use the .from_raster_xarray method instead of my custom method for my hazard file without running into memory issues 👍
My hazard is a xarray dataset created from multiple netcdf files, and the following dimensions: time: 3660, chy: 225, chx: 352

[timschmi95]

This change does enable to use the .from_raster_xarray method instead of my custom method for my hazard file without running into memory issues 👍 My hazard is a xarray dataset created from multiple netcdf files, and the following dimensions: time: 3660, chy: 225, chx: 352

Actually, I realize that depending on the previous memory allocation I still run into a memory error. I'm not sure if the chunking could be made more efficient, or if it's just too much for my laptop anyways and I either pre-slice the data to a smaller extent (like in my custom function), or run it on euler if I need the full extent.

[peanutfun]

Actually, I realize that depending on the previous memory allocation I still run into a memory error.

Xarray does not check how much memory in your system is already in use. I currently set chunks="auto", which simply ensures that one chunk can fit into the entire system RAM. However, from_raster_xarray explicitly supports passing Datasets that are already opened, so you can open it with custom chunking. The following would be a somewhat conservative choice:

ds = xr.open_dataset("file.nc", chunks=dict(time=100, chy=-1, chx=-1))
hazard = Hazard.from_raster_xarray(ds, **kwargs)

[peanutfun]

@emanuel-schmid The sparse package still seems to be missing from the testing environment

[emanuel-schmid]

@emanuel-schmid The sparse package still seems to be missing from the testing environment

rather again than still ;)
But it's been re-installed just now.

[emanuel-schmid]

tests are running and I'm gonna include sparse in the develop branch

[emanuel-schmid]

Let me quickly come back to a topic related to a previous PR, not really related to this one but by the method concerned:
Why did we chose 'time' as the default 'event' coordinate name, and not 'event'? 🧐

[peanutfun]

Why did we chose 'time' as the default 'event' coordinate name, and not 'event'?

@emanuel-schmid In the Hazard class, we distinguish certain events, that can occur at the same or different times. However, most hazard datasets contain a time variable, but not an event variable. So the choice was made to use the most reasonable default value, but to still call the key event, and not time, to make clear that the variable will relate to different events in the Hazard class context. Does that make sense? 😵

[emanuel-schmid]

Yes, that makes sense. 👍 I've added a note to this respect in the pydoc string in case somebody else is wondering. Feel free to revert if it isn't appropriate.

[peanutfun]

Very good! 🤩

[peanutfun]

@emanuel-schmid Is this ready to go from your perspective?

[emanuel-schmid]

🎉 Ready to go, from my perspective.

[emanuel-schmid]

🙇‍♂️ many thanks!