Complete dask conversion of XArrayResamplerBilinear

[pnuu]

The xarray/dask version of bilinear resampling works, but much of the data are computed too early and in-memory. The "daskification" of the class methods should be completed.

[pnuu]

I've been taking a deep look into this, and my conclusions are:

• changing calls from np.func to da.func doesn't give any further benefit
• ... until dask can do fancy indexing (e.g. here: https://github.com/pytroll/pyresample/blob/master/pyresample/bilinear/xarr.py#L419)
• memory usage could be lowered quite significantly (with my test area from 8 GB to 5 GB) by writing np.transposte(np.tile() as

def _tile(arr, repeat):
    lst = []
    for _ in range(repeat[0]):
        lst.append(arr)
    return da.stack(lst, axis=1) 

Dask profiling with SatPy shows that when using on-disk caching for the resampling LUTs, only the first run (where the LUTs are calculated) is slow, after that both the memory usage and overall processing time is very low. With my test (SEVIRI, one RGB composite, target area of 1959 x 1934 pixels) the memory use is reduced from 8 GB to less than 2 GB, and processing time went down from 52 s to 6.5 s.

[rabernat]

Is the conclusion of this discussion that pyresample will not support lazy resampling of dask arrays?

[djhoese]

@rabernat No, support exists. Much of pyresample is dask/xarray friendly but not easily accessible from pyresample's traditional interfaces. We currently have a terrible spread of resampling functionality across satpy and pyresample. It all uses xarray and dask. We have plans of making a pyresample 2.0 with better interfaces for both numpy-based workflows and dask workflows (probaby an xarray accessor). This issue is specifically about bilinear resampling which is relatively new and hasn't been profiled as aggressively as the nearest neighbor resampling.

Additionally we have plans for a gradient search resampling (see #191) that would be faster than the current nearest neighbor and also use dask, but may be using dask in unintended ways. @mraspaud could provide more information if you're curious.

If you have something you'd like to do specifically let me know and we can discuss it here or on the pangeo gitter.

[rabernat]

Thanks for the reply @djhoese. I opened a new issue about my use case.

[pnuu]

Superseded by #215