Function in xarray to regroup monthly data into months and # of years

[pecos27]

I have a 3D array with time (in months), y, and x. I'm hoping to split this xarray dataframe into # of months (january - december), all the years of that month, y, and x.
For example, I have 36 years worth of monthly data, and I would like to reorder the xarray dataframe as:
months (1 - 12), # of years of that month, y, and x. So for the month of april, the dataframe would be 1, 36 (36 years of only april), y, and x.
How might I do that in xarray?

I tried using:

split_months = ds.groupby('time.month')

however it appears to return just an empty dataframe:

DataArrayGroupBy, grouped over 'month' 12 groups with labels 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12..
I get a typical xarray dataframe returned when I do

split_months = ds.groupby('time.month').mean('time') however I do not want to take the average or sum over all the years, I just want to split the data into months, # of years of that month only, y, and x.

Is it possible to do this in xarray?

[chiaral]

you can iterate across the group

for month_id, ds_month in ds.groupby('time.month'):
    print(month_id, ds_month)

you could create a new dataset with another dimension (year) through this loop, or you could simply apply your function to each month separately.

[chiaral]

this
split_months = ds.groupby('time.month')
is not giving you an empty dataframe. split_months is simply not a dataframe. it's a groupby object, in which data are grouped.

[pecos27]

Thank you! yeah... I tried making it a list at first. How would you make a new dataset with another dimension in that loop?
Something along the lines of this maybe...?
new_data_set = [] for month_id, ds_month in ds.groupby('time.month'): print(month_id, ds_month) new_data_set.append(ds[month_id,ds_month])

[pecos27]

Sorry, just one more question. I tried doing this (making a new data array within the loop):

`for month_id, ds_month in ds.groupby('time.month'):

print(month_id, ds_month)

new = xr.DataArray(month_id,ds_month)`

but I get an error that says "maximum supported dimension for an ndarray is 32, found 40". Do you know a way to work around this? Thank you!

[dcherian]

One option is ds.rolling(time=12).construct(window_dim="month").isel(...)

See #2419 for an alternative way.

I think the real solution is to support .coarsen(time=12).construct(dim="month")

[pecos27]

thank you! I tried the ds.rolling first, but I get an error with the chunk size... Sorry I am new to xarray. Do you know what that error might be from?

For window size 12, every chunk should be larger than 6, but the smallest chunk size is 1. Rechunk your array with a larger chunk size or a chunk size that more evenly divides the shape of your array.

[dcherian]

can you update to latest xarray?

[pecos27]

Yeah! I just updated. now it's version 0.17.0. I'm still getting the chunks error unfortunately... Could it be the way I am loading in the data potentially? I am using EN4 analyses (salinity and temperature spatial data) using: ds=xr.open_mfdataset('en4 analyses').

[dcherian]

Oh sorry, my mistake. We haven't released a version that relaxes that restriction. It's saying that you have one timestep per chunk. For this to work use .chunk({"time": 12}) before the rolling call. You might have to adjust chunk size along the other dimensions to compensate.

[mathause]

Does the following snippet do what you want?

def to_monthly(ds):
    year = ds.time.dt.year
    month = ds.time.dt.month

    # assign new coords
    ds = ds.assign_coords(year=("time", year.data), month=("time", month.data))

    # reshape the array to (..., "month", "year")
    return ds.set_index(time=("year", "month")).unstack("time")  

Example

# read tutorial data and resample to monthly
air = xr.tutorial.open_dataset("air_temperature").resample(time="M").mean()

to_monthly(air)

Out[8]: 
<xarray.Dataset>
Dimensions:  (lat: 25, lon: 53, month: 12, year: 2)
Coordinates:
  * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
  * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
  * year     (year) int64 2013 2014
  * month    (month) int64 1 2 3 4 5 6 7 8 9 10 11 12
Data variables:
    air      (lat, lon, year, month) float32 244.5 240.7 248.3 ... 298.8 297.7

---

edit: use year.data in assign_coords (instead of just year) for compatibility with newer versions of xarray.

[pecos27]

This is great! Thank you so much

[kuchaale]

If I try this approach, I get the following error:

[mathause]

You are right, there was a mistake in my example. It has to be:

air = xr.tutorial.open_dataset("air_temperature").resample(time="M").mean()

The mean was missing - it's now corrected above.

[awais307]

Thanks. I am trying a similar approach for a slightly different outcome.

I want to resample monthly time series which looks like;

qtot_y
Out[243]:
<xarray.Dataset>
Dimensions:  (lat: 360, lon: 720, month: 4, year: 86)
Coordinates:
  * lon      (lon) float32 -179.8 -179.2 -178.8 -178.2 ... 178.8 179.2 179.8
  * lat      (lat) float32 89.75 89.25 88.75 88.25 ... -88.75 -89.25 -89.75
  * year     (year) int64 2010 2011 2012 2013 2014 ... 2095 2096 2097 2098 2099
  * month    (month) int64 1 4 7 10
Data variables:
    qtot     (lat, lon, year, month) float64 dask.array<chunksize=(360, 720, 41, 4), meta=np.ndarray>

to 5 year time steps such as;
2010, 2015, 2020 with same monthly times.

I try to do

qtot_y = qtot_y.resample(time ='5Y').mean()
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\salenv2\lib\site-packages\xarray\core\dataset.py", line 1259, in _construct_dataarray
    variable = self._variables[name]
KeyError: 'time'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<ipython-input-244-0dbe69009443>", line 1, in <module>
    qtot_y = qtot_y.resample(time ='5Y').mean()
  File "C:\ProgramData\Anaconda3\envs\salenv2\lib\site-packages\xarray\core\common.py", line 1121, in resample
    dim_coord = self[dim]
  File "C:\ProgramData\Anaconda3\envs\salenv2\lib\site-packages\xarray\core\dataset.py", line 1359, in __getitem__
    return self._construct_dataarray(key)
  File "C:\ProgramData\Anaconda3\envs\salenv2\lib\site-packages\xarray\core\dataset.py", line 1261, in _construct_dataarray
    _, name, variable = _get_virtual_variable(
  File "C:\ProgramData\Anaconda3\envs\salenv2\lib\site-packages\xarray\core\dataset.py", line 171, in _get_virtual_variable
    ref_var = variables[ref_name]
KeyError: 'time'

Any solutions to this ?

[pecos27]

Hi! Is there a way to rebuild the time series by restacking the month and years in chronological order? @mathause
Basically do the reverse of the function to_monthly and retrieve the original time series.

I was thinking something along the lines of:

def reorder(ds):
    year = ds.year
    month = ds.month

    # assign new coords
    #ds = ds.assign_coords(month=("time", month),year=("time", year))

    # reshape the array to (..., "month", "year")
    return ds.set_index(time=("month", "year")).stack("time") 

Thank you so much!