Dask based implemenetation of reproject #88
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Since numpy arrays shape is of type Tuple[int, ...] mark those types as acceptable inputs into shape construction function. ValueError will be raised at runtime when input is not 2-tuple.
This allows working with arbitrary 2d Dask inputs, and not just those that have same shape of tiles except for edge tiles
signature was missing chunk information
Call some function with `[]` instead of `()`, handy for exposing slicing into some object. `(x: TK)->TV` into `[x: TK]->TV`.
Support slices on top of individual indexes Also introduce `.crop` functionality.
Declare returned type as just (slice, slice), was (NormalizedSlice, NormalizedSlice). This is to be type compatible with numpy.
This will be used in dask reproject to assemble computed chunks into a contigous array.
x[-1,:] and x[-1:,:] should behave the same, but x[-1,:] case was broken.
From a set of tile indexes compute cropped version and map indexes into the cropped space.
support yxb/tyxb axis order on input
- not fully tested yet - todo: share intersection work across multiple bands of a single dataset that share common chunking regime
There were still assumptions about fixed sized chunks in the code, so that broke for variable-sized chunks Adding `.locate(pix) -> tile` to RoiTiles classes and using that to map from world geometry to tile indexes.
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## develop #88 +/- ##
===========================================
- Coverage 98.11% 96.51% -1.61%
===========================================
Files 23 25 +2
Lines 3819 4188 +369
===========================================
+ Hits 3747 4042 +295
- Misses 72 146 +74
☔ View full report in Codecov by Sentry. |
Merged
3 tasks
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
GeoboxTilesGCPGeoBoxinputGeoboxTiles.grid_intersectmethod for computing cross-block depenendenciesBlockAssemblerhelper classCloses #26
TODO:
grid_intersectresults when transforming multi-band Daskxarray.Datasets that share common geometry of chunks across several bands to speed up Dask graph construction