Dask outputs warning: "The da.atop function has moved to da.blockwise"

[lumbric]

Problem description

dask 1.1.0 moved atop() to blockwise() and introduced a warning when atop() is used.

Related

• upstream ticket and PR of dask change: Move dask/array/top.py code to dask/blockwise.py dask/dask#4348 Better name for atop? dask/dask#4035
• the warning in the dask documentation in an xarray example, probably not on purpose
• warnings have been already discussed in silence a couple of warnings in tests #2727, but not fixed there
• same issue in a different project: Update to fix deprecation warning from dask regarding atop pytroll/satpy#608

Code Sample

import numpy as np
import xarray as xr

xr.DataArray(np.ones(1000))
d = xr.DataArray(np.ones(1000))
d.to_netcdf('/tmp/ones.nc')
d = xr.open_dataarray('/tmp/ones.nc', chunks=10)
xr.apply_ufunc(lambda x: 42 * x, d, dask='parallelized', output_dtypes=[np.float64])

This outputs the warning:

...lib/python3.7/site-packages/dask/array/blockwise.py:204: UserWarning: The da.atop function has moved to da.blockwise
  warnings.warn("The da.atop function has moved to da.blockwise")

Expected Output

No warning. As user of a recent version of dask and xarray, there shouldn't be any warnings if everything is done right. The warning should be tackled inside xarray somehow.

Solution

Not sure, can xarray break compatibility with dask <1.1.0 with some future version? Otherwise I guess there needs to be some legacy code in xarray which calls the right function.

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.7.3 (default, Mar 27 2019, 22:11:17) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.18.0-17-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.1

xarray: 0.12.1
pandas: 0.24.2
numpy: 1.16.3
scipy: 1.2.1
netCDF4: 1.4.2
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.0.3.4
nc_time_axis: None
PseudonetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 1.2.0
distributed: 1.27.0
matplotlib: 3.0.3
cartopy: None
seaborn: 0.9.0
setuptools: 41.0.0
pip: 19.1
conda: None
pytest: 4.4.1
IPython: 7.5.0
sphinx: None

[shoyer]

Not sure, can xarray break compatibility with dask <1.1.0 with some future version? Otherwise I guess there needs to be some legacy code in xarray which calls the right function.

Thanks for the report.

Yes, we can definitely do this in the future, but for now, I think we should try to be backwards compatible.

Using something like the following instead xarray should solve this:

try:
    blockwise = dask.array.blockwise  # new name in dask 1.1
except AttributeError:
    blockwise = dask.array.atop

Any interest in putting together a PR?

[lumbric]

Any interest in putting together a PR?

Yes, can do so. When writing the report, I actually thought maybe preparing a PR is easier to write and to read than the ticket... :) In this case it really shouldn't be a big deal fixing it.

Maybe a bit off-topic: The thing I don't really understand and why I wanted to ask first: is there a clear paradigm about compatibility in the pydata universe? Despite its 0.x version number, I guess xarray tries to stay backward compatible regarding its public interface, right? When are the versions of dependencies increase? Simply motivated by need of new features in one of the dependent libraries?

[shoyer]

The thing I don't really understand and why I wanted to ask first: is there a clear paradigm about compatibility in the pydata universe? Despite its 0.x version number, I guess xarray tries to stay backward compatible regarding its public interface, right? When are the versions of dependencies increase? Simply motivated by need of new features in one of the dependent libraries?

The usual rule of thumb is to keep things working for at least one "major" release with a deprecation warning, so that somebody who continuously runs their code has time to notice before things break.

The exact appropriate time to wait is project dependent. In xarray's case, we've usually been waiting 6-12 months before making breaking changes. NumPy might wait 2-3 years.

There isn't a strict rule for bumping required dependency versions, but again it's motivated by a desire to give users some flexibility so as not to require them to upgrade everything at once.

[lumbric]

Fixed in 714ae86.

(Sorry for the delay... I actually prepared a PR but never finished it completely even it was such a simple thing.)