Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grouped operations (and more) #46

Merged
merged 20 commits into from
Jan 22, 2024
Merged

Grouped operations (and more) #46

merged 20 commits into from
Jan 22, 2024

Conversation

valpesendorfer
Copy link
Member

This PR came from improvement work done for pre-calculating SPIs and such for admin extractions.

Most notably this PR adds:

  • computation of groups along time dimension
    • gammastd_grp for grouped SPIs, that can be calculated by passing an array with group indices to the .algo.spi accessor
    • mean_grp grouped means (accessible through algo.mean_grp accessor)
  • a new accessor rolling that allows for rolling computations over a defined window. Currently, rolling.sum is implemented
  • a fix for zonal.mean which prevented the definition of output datatype

@@ -6,7 +6,7 @@


@lazycompile(njit)
def do_mean(pixels, z_pixels, num_zones, nodata, z_nodata, dtype="float64"):
def do_mean(pixels, z_pixels, num_zones, nodata, z_nodata, out_dtype=np.float32):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this change is necessary? don't see any matching call-site changes

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I see why. Usually this problem of passing extra args to dask graph is best done with functools.partial(my_func, **fixed_named_args_that_might_alias)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There were a couple of problems with the previous implementation:

  • dtype is already a keyword to map_blocks, so the naming is ambiguous when passed in through the accessor (which could be done with functools.partial I suppose.
  • second problem was numba: when passing in a dtype with string notation it would fail compiling. Not sure if that would have been solved by using functools.partial, but passing the object works fine

@valpesendorfer
Copy link
Member Author

OK this is now ready to be merged.

Quick summary:

Additions

  • computation of groups along time dimension
    • gammastd_grp for grouped SPIs, that can be calculated by passing an array with group indices to the .algo.spi accessor
    • mean_grp grouped means (accessible through algo.mean_grp accessor)
  • a new accessor rolling that allows for rolling computations over a defined window. Currently, rolling.sum is implemented

Improvements

  • better handling of nodata in SPI methods
  • a fix for zonal.meanwhich prevented the definition of output datatype

Unrelated changes

  • surface Dekad.ndays from hdc.algo.dekad.Dekad to the Period accessor of hdc.algo

hdc/algo/ops/stats.py Outdated Show resolved Hide resolved
@valpesendorfer
Copy link
Member Author

Updated summary:

Additions

  • computation of groups along time dimension
    • gammastd_grp for grouped SPIs, that can be calculated by passing an array with group indices to the .algo.spi accessor
    • mean_grp grouped means (accessible through algo.mean_grp accessor)
  • a new accessor rolling that allows for rolling computations over a defined window. Currently, rolling.sum is implemented

Improvements

  • better handling of nodata in SPI methods
  • a fix for zonal.meanwhich prevented the definition of output datatype

Unrelated changes

  • surface from hdc.algo.dekad.Dekad to the Period accessor of hdc.algo:
    • Dekad.ndays
    • Dekad.start_date
    • Dekad.end_date
    • Dekad.raw

@valpesendorfer valpesendorfer merged commit 15c0297 into main Jan 22, 2024
12 checks passed
@valpesendorfer valpesendorfer deleted the spi-updates branch January 22, 2024 19:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants