Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to specify chunks to iris.util.broadcast_to_shape #5456

Closed
bouweandela opened this issue Sep 1, 2023 · 0 comments · Fixed by #5620
Closed

Add the ability to specify chunks to iris.util.broadcast_to_shape #5456

bouweandela opened this issue Sep 1, 2023 · 0 comments · Fixed by #5620
Assignees

Comments

@bouweandela
Copy link
Member

✨ Feature Request

I would like to be able to specify the target chunks in iris.util.broadcast_to_shape.

Motivation

This function is often used to broadcast auxiliary coordinate or cell measure data or weights to the same shape as the data from a cube. If the chunks of the broadcasted data do not match that of the cube, this leads to poor computational performance.

Additional context

Here is an example implementation (taken from ESMValGroup/ESMValCore#2120):

def broadcast_to_shape(array, shape, chunks, dim_map):
    """Copy of `iris.util.broadcast_to_shape` that allows specifying chunks."""
    if isinstance(array, da.Array):
        chunks = list(chunks)
        for src_idx, tgt_idx in enumerate(dim_map):
            # Only use the specified chunks along new dimensions.
            chunks[tgt_idx] = array.chunks[src_idx]
        broadcast_to = partial(da.broadcast_to, chunks=chunks)
    else:
        broadcast_to = np.broadcast_to

    n_orig_dims = len(array.shape)
    n_new_dims = len(shape) - n_orig_dims
    array = array.reshape(array.shape + (1,) * n_new_dims)

    # Get dims in required order.
    array = np.moveaxis(array, range(n_orig_dims), dim_map)
    new_array = broadcast_to(array, shape)

    if np.ma.isMA(array):
        # broadcast_to strips masks so we need to handle them explicitly.
        mask = np.ma.getmask(array)
        if mask is np.ma.nomask:
            new_mask = np.ma.nomask
        else:
            new_mask = np.broadcast_to(mask, shape)
        new_array = np.ma.array(new_array, mask=new_mask)

    elif is_lazy_masked_data(array):
        # broadcast_to strips masks so we need to handle them explicitly.
        mask = da.ma.getmaskarray(array)
        new_mask = broadcast_to(mask, shape)
        new_array = da.ma.masked_array(new_array, new_mask)

    return new_array
@bouweandela bouweandela moved this to 🆕 New in ESMValTool Sep 1, 2023
@scitools-ci scitools-ci bot removed this from 🚴 Peloton Dec 15, 2023
@trexfeathers trexfeathers moved this from 🆕 New to 🏗 In progress in ESMValTool Feb 19, 2024
@trexfeathers trexfeathers assigned pp-mo and HGWright and unassigned pp-mo Feb 29, 2024
@trexfeathers trexfeathers moved this from 🏗 In progress to 👀 In review in ESMValTool Mar 1, 2024
@github-project-automation github-project-automation bot moved this from 👀 In review to ✅ Done in ESMValTool Apr 5, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in ESiWACE3 ESMValTool service Apr 5, 2024
@scitools-ci scitools-ci bot removed this from 🚴 Peloton May 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants