Many methods are broken (e.g., concat/stack/sortby) when using repeated dimension names

[fsteinmetz]

Concatenating DataArrays with repeated dimensions does not work.

import xarray as xr  #xarray 0.8.2
from numpy import eye
A = xr.DataArray(eye(3), dims=['dim0', 'dim0'])
xr.concat([A, A], 'newdim')

fails with

[...]
ValueError: axes don't match array

[fmaussion]

Yes, also happening on latest master.

I suspect there are several other things which won't work properly (or at least unexpectedly) when having repeated dims...

[shoyer]

Indeed, we don't have very good test coverage for operations with repeated dimensions. Fixes would certainly be appreciated, though they might be somewhat tricky. Even failing loudly with ValueError: repeated dimensions not yet supported would be an improvement over the current state.

[fsteinmetz]

Right, also positional indexing works unexpectedly in this case, though I understand it's tricky and should probably be discouraged:

A[0,:]  # returns A
A[:,0]  # returns A.isel(dim0=0)

[fmaussion]

I guess it would be good to document the expected behaviour with repeated dims somewhere? I.e. what should happen when doing:

a = xr.DataArray(eye(3), dims=['dim0', 'dim0'])
a.mean(dim='dim0')

?

[fsteinmetz]

I cannot see a use case in which repeated dims actually make sense.

In my case this situation originates from h5 files which indeed contains repeated dimensions (variables(dimensions): uint16 B0(phony_dim_0,phony_dim_0), ..., uint8 VAA(phony_dim_1,phony_dim_1)), thus xarray is not to blame here.
These are "dummy" dimensions, not associated with physical values. What we do to circumvent this problem is "re-dimension" all variables.
Maybe a safe approach would be for open_dataset to raise a warning by default when encountering such variables, with possibly an option to perform automatic or custom dimension naming to avoid repeated dims.
I also agree with @shoyer that failing loudly when operating on such DataArrays instead of providing confusing results would be an improvement.

[fmaussion]

In my case this situation originates from h5 files which indeed contains repeated dimensions

Yes this happened to me too. First thing I did is converting the files to proper netcdf datasets...

[shoyer]

I cannot see a use case in which repeated dims actually make sense.

Agreed. I would have disallowed them entirely, but sometimes it's useful to allow loading variables with duplicate dimensions, even if the only valid operation you can do is de-duplicate them.

Every routine that looks up dimensions by name should go through the get_axis_num method. That would be a good place to add a check for uniqueness.

[gerritholl]

I cannot see a use case in which repeated dims actually make sense.

I use repeated dimensions to store a covariance matrix. The data variable containing the covariance matrix has 4 dimensions, of which the last 2 are repeated. For example, I have a data variable with dimensions (channel, scanline, element, element), storing an element-element covariance matrix for every scanline in satellite data.

This is valid NetCDF and should be valid in xarray. It would be a significant problem for me if they became disallowed.

[jhamman]

@gerritholl - rereading this issue, I don't think we're particularly opposed to supporting duplicate dimensions. We do know there are things that don't work right now and that we don't have test coverage for operations that use duplicate dimensions.

This is marked as a help wanted issue and I suspect that if someone like yourself, who has a use case for this functionality, were to want to work on this issue, we'd be happy to see it move forward.

[gerritholl]

@jhamman Ok, good to hear it's not slated to be removed. I would love to work on this, I wish I had the time! I'll keep it in mind if I do find some spare time.

[gerritholl]

This also affects the stack method.

[Hoeze]

Annotating distance matrices with xarray is not possible as well due to the duplicate dimension.