DEP: missing explicit dependency declaration (datatree)

[neutrinoceros]

• Py-ART version: 1.19.1
• Python version: 3.10
• Operating System: Linux

Description

installing pyart in a fresh env appears to be currently broken, because some modules depend on datatree, which is not declared as an explicit dependency.
I'm assuming this is an oversight from #1622

What I Did

$ python -m venv
$ source .venv/bin/activate
$ python -m pip install arm-pyart
$ python -c "from pyart import io"
## You are using the Python ARM Radar Toolkit (Py-ART), an open source
## library for working with weather radar data. Py-ART is partly
## supported by the U.S. Department of Energy as part of the Atmospheric
## Radiation Measurement (ARM) Climate Research Facility, an Office of
## Science user facility.
##
## If you use this software to prepare a publication, please cite:
##
##     JJ Helmus and SM Collis, JORS 2016, doi: 10.5334/jors.119

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/tmp/.venv/lib/python3.12/site-packages/pyart/__init__.py", line 29, in <module>
    from . import (
  File "/private/tmp/.venv/lib/python3.12/site-packages/pyart/xradar/__init__.py", line 1, in <module>
    from .accessor import Xradar, Xgrid  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/.venv/lib/python3.12/site-packages/pyart/xradar/accessor.py", line 8, in <module>
    import datatree
ModuleNotFoundError: No module named 'datatree'

[neutrinoceros]

I also note that datatree was last released in 2011 and that I can't build it on Apple Silicon, so I'd like to ask whether it'd be possible to not depend on it at all ?

[kmuehlbauer]

@neutrinoceros I can't speak for the maintainers here, but datatree is a major building block of the new CfRadial2/FM301 based datamodel pyart will eventually converge to. The former experimental implementation in https://github.com/xarray-contrib/datatree (which has a recent 0.0.15 version) has been merge/transformed into xarray codebase as of xarray 2024.10.0

[neutrinoceros]

as long as there's a portable solution I'm happy to use it

[kmuehlbauer]

Yes, be sure to use https://pypi.org/project/xarray-datatree/ not https://pypi.org/project/datatree/.

[neutrinoceros]

note that adding

diff --git a/requirements.txt b/requirements.txt
index cec08287..edde1d57 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -13,3 +13,4 @@ mda-xdrlib
 xarray
 cartopy
 pint
+xarray-datatree

I still get an error when running python -c "from pyart import io"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/clm/dev/pyart/pyart/__init__.py", line 29, in <module>
    from . import (
  File "/Users/clm/dev/pyart/pyart/xradar/__init__.py", line 1, in <module>
    from .accessor import Xradar, Xgrid  # noqa
  File "/Users/clm/dev/pyart/pyart/xradar/accessor.py", line 8, in <module>
    import datatree
  File "/Users/clm/dev/pyart/.venv/lib/python3.10/site-packages/datatree/__init__.py", line 2, in <module>
    from .datatree import DataTree
  File "/Users/clm/dev/pyart/.venv/lib/python3.10/site-packages/datatree/datatree.py", line 33, in <module>
    from xarray.core.utils import (
ImportError: cannot import name 'HybridMappingProxy' from 'xarray.core.utils' (/Users/clm/dev/pyart/.venv/lib/python3.10/site-packages/xarray/core/utils.py)

[kmuehlbauer]

See yt-project/yt#5041

[juhi24]

Heads up: xarray-datatree has been merged to xarray since 2024.10.0. Related: #1681

[mgrover1]

I agree with @kmuehlbauer 's points here! I think the best steps are:

• pin xradar<=0.7.0 and xarray<=2024.9.0 and do a release (FIX: Fix pinning of xradar and xarray #1682)
• fix anything which has to be fixed (FIX: Ensure backwards compatibility with datatree/new versions of xarray #1681)
• pin xradar>=0.8.0 and xarray>=2024.10.0 and do a another release

[mgrover1]

I agree with @kmuehlbauer 's points here! I think the best steps are:

• pin xradar<=0.7.0 and xarray<=2024.9.0 and do a release (FIX: Fix pinning of xradar and xarray #1682)
• fix anything which has to be fixed (FIX: Ensure backwards compatibility with datatree/new versions of xarray #1681)
• pin xradar>=0.8.0 and xarray>=2024.10.0 and do a another release

This should all be resolved with the release yesterday, and the release today. The pypi wheels are currently building, and the conda-forge version of the updated xradar/xarray versions will be out by the end of the day today. Thank you for raising this critical issue @neutrinoceros @juhi24 and for the suggestions/feedback here, as well as upstream fixes @kmuehlbauer ❤️

[neutrinoceros]

Thank you !