[WIP] apply_gufunc

[TomNicholas]

WIP - all I've really done so far is copy across the relevant dask code from dask/array/gufunc.py. It's really just copied verbatim, but not importable as it's not factored out into smaller functions in dask.

Comments:

• Does copying code verbatim present a licensing issue? Is there a better approach?
• We might want an asarray function
• Do we want to copy dask's whole meta thing?
• blockwise will need concatenate kwarg
• dask has a dependency on toolz that I haven't added here yet
• I don't yet understand what the last part of dask.array.apply_gufunc is doing, the highlevel graph construction. I need to look through that more closely.

[tomwhite]

Awesome - thanks for starting on this @TomNicholas!

• Does copying code verbatim present a licensing issue? Is there a better approach?

I don't think it necessarily is an issue, but we should provide attribution, much like Dask does for NumPy for example: https://github.com/dask/dask/blob/cf7781bcf2c7b478b701fd848e78d51c5a9c6e8f/dask/array/core.py#L4071-L4073.

Longer-term it would be interesting to see if it's possible to share code, perhaps by taking advantage of Dask's HLG work - but I'm not very familiar with that.

• We might want an asarray function

There's one in https://github.com/tomwhite/cubed/blob/main/cubed/array_api/creation_functions.py

• Do we want to copy dask's whole meta thing?

There's no meta in Cubed, and so far it's been OK to pass dtype, shape, chunks etc around directly. Would that work here?

• blockwise will need concatenate kwarg

map_blocks in Dask uses concatenate=True, but Cubed doesn't so maybe follow the same approach if possible. Also,
IIRC concatenate can cause memory issues (e.g. in Dask matmul it was removed), so another reason to avoid if we can.

• dask has a dependency on toolz that I haven't added here yet

I think we use toolz in other places already, so that's not a problem.

• I don't yet understand what the last part of dask.array.apply_gufunc is doing, the highlevel graph construction. I need to look through that more closely.

It might be helpful to see how it was implemented before HLGs were introduced, just to help get some understanding.

[tomwhite]

It would be useful to add support for gufuncs for the Pangeo examples. Is this PR something you would like to continue working on @TomNicholas? I could look at it in the new year if not.

[TomNicholas]

Hey Tom, Sorry, as much as I would love to be working on this, realistically I'm not going to get to it for another month at least. Don't let me stop you pushing things forward in the meantime!

…

On Mon, Dec 19, 2022, 7:57 AM Tom White ***@***.***> wrote: It would be useful to add support for gufuncs for the Pangeo examples. Is this PR something you would like to continue working on @TomNicholas <https://github.com/TomNicholas>? I could look at it in the new year if not. — Reply to this email directly, view it on GitHub <#119 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AISNPI7UBEGSFRNWE2M6KPLWOBLTXANCNFSM6AAAAAAQDTNNIU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tomwhite]

I plan to work on this soon.

[tomwhite]

I'm going to close this now, as this was superseded by #149. Thanks for starting this @TomNicholas - it really helped me with the work in #149.