torchdata not compatible with torch 2.3.0

[mmdanziger]

🐛 Describe the bug

Describe the bug

torchdata does not work with torch 2.3.0 because DILL_AVAILABLE is not available where expected:

Python 3.12.2 (main, Feb  6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchdata
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/__init__.py", line 9, in <module>
    from . import datapipes
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/datapipes/__init__.py", line 9, in <module>
    from . import iter, map, utils
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/datapipes/iter/__init__.py", line 79, in <module>
    from torchdata.datapipes.iter.util.cacheholder import (
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/datapipes/iter/util/cacheholder.py", line 24, in <module>
    from torch.utils.data.datapipes.utils.common import _check_unpickable_fn, DILL_AVAILABLE
ImportError: cannot import name 'DILL_AVAILABLE' from 'torch.utils.data.datapipes.utils.common' (/private/tmp/test/lib/python3.12/site-packages/torch/utils/data/datapipes/utils/common.py)
>>>

This was fixed in pytorch/pytorch#122616 but the fix did not make it into the torch 2.3.0 release
https://github.com/pytorch/pytorch/blob/v2.3.0/torch/utils/data/datapipes/utils/common.py

I would assume 2.3.1 will be ok because the fix is in main. For now though, a clean install of torchdata is broken.

I would recommend adding a torch>=2,!=2.3.0 to setup.py

To Reproduce

pip install torchdata
import torchdata

Expected behavior

successful import

Environment

fresh install

Versions

Collecting environment information...
PyTorch version: 2.3.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.4.1 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.3.9.4)
CMake version: version 3.27.1
Libc version: N/A

Python version: 3.12.2 (main, Feb 6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)] (64-bit runtime)
Python platform: macOS-14.4.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.3.0
[pip3] torchdata==0.7.1
[conda] Could not collect

[Rhett-Ying]

I encountered the same issue. Is there any ETA to merge the fix into torch 2.3?

[h-vetinari]

We'd also need this as soon as feasible for rolling out pytorch 2.3 support in conda-forge.

[atalman]

This is duplicate of pytorch/pytorch#124907