Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ValueError: DataLoader with IterableDataset: expected unspecified shuffle option #1576

Closed
KumoLiu opened this issue Nov 19, 2023 · 0 comments · Fixed by #1577
Closed

ValueError: DataLoader with IterableDataset: expected unspecified shuffle option #1576

KumoLiu opened this issue Nov 19, 2023 · 0 comments · Fixed by #1577
Labels
bug Something isn't working

Comments

@KumoLiu
Copy link
Contributor

KumoLiu commented Nov 19, 2023

[2023-11-19T03:55:52.973Z] ---------------------------------------------------------------------------
[2023-11-19T03:55:52.973Z] Exception encountered at "In [7]":
[2023-11-19T03:55:52.973Z] ValueError Traceback (most recent call last)
[2023-11-19T03:55:52.973Z] Cell In[7], line 31
[2023-11-19T03:55:52.973Z] 17 patch_transform 241m=ompose(
[2023-11-19T03:55:52.973Z] 18 [
[2023-11-19T03:55:52.973Z] 19 SqueezeDimd(keys;5;241m=124m"38;5;124mimgm;5;124m"124m"38;5;124msegm;5;124m" dim;5;241m=38;5;241m-38;5;241m1m), 66;03m# squeeze the last dim;00m
[2023-11-19T03:55:52.973Z] (...)023-11-19T03:55:52.973Z] 23 ]
[2023-11-19T03:55:52.973Z] 24 )
[2023-11-19T03:55:52.973Z] 25 patch_ds 241m=atchDataset(
[2023-11-19T03:55:52.973Z] 26 volume_ds,
[2023-11-19T03:55:52.973Z] 27 transform;5;241m=tch_transform,
[2023-11-19T03:55:52.973Z] 28 patch_func;5;241m=tch_func,
[2023-11-19T03:55:52.973Z] 29 samples_per_image;5;241m=m_samples,
[2023-11-19T03:55:52.973Z] 30 )
[2023-11-19T03:55:52.973Z] ---> 31 train_loader 241m=taLoadermm(2023-11-19T03:55:52.973Z] 32 43mpatch_dsmm,2023-11-19T03:55:52.973Z] 33 43mbatch_sizem;5;241;43m=m;5;241;43m3;49mm,2023-11-19T03:55:52.973Z] 34 43mshufflem;5;241;43m=m;5;28;43;01mTrue;49;00mm,43m 38;5;66;43;03m# this shuffles slices from different volumes;49;00m
[2023-11-19T03:55:52.973Z] 35 43mnum_workersm;5;241;43m=m;5;241;43m2;49mm,2023-11-19T03:55:52.973Z] 36 43mpin_memorym;5;241;43m=mmtorchm;5;241;43m.mmcudam;5;241;43m.mmis_availablemm(43m)43m,2023-11-19T03:55:52.973Z] 37 2023-11-19T03:55:52.973Z] 38 check_data 241m=onai;5;241m.ils;5;241m.sc;5;241m.rst(train_loader)
[2023-11-19T03:55:52.973Z] 39 28mprintm(124m"38;5;124mfirst patchm;5;124m'38;5;124ms shape: 38;5;124m"check_data[124m"38;5;124mimgm;5;124m"241m.ape, check_data[124m"38;5;124msegm;5;124m"241m.ape)
[2023-11-19T03:55:52.973Z]
[2023-11-19T03:55:52.973Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/data/dataloader.py:106, in DataLoader.__init__34m(self, dataset, num_workers, **kwargs)023-11-19T03:55:52.973Z] 94 28;01mif;00m (
[2023-11-19T03:55:52.973Z] 95 124m"38;5;124mmultiprocessing_contextm;5;124m"129;01min;00m kwargs
[2023-11-19T03:55:52.973Z] 96 129;01mand;00m kwargs[124m"38;5;124mmultiprocessing_contextm;5;124m"241m==124m"38;5;124mspawnm;5;124m"2023-11-19T03:55:52.973Z] 97 129;01mand;00m 129;01mnot;00m get_track_meta()
[2023-11-19T03:55:52.973Z] 98 ):
[2023-11-19T03:55:52.973Z] 99 warnings;5;241m.rn(
[2023-11-19T03:55:52.973Z] 100 124m"38;5;124mPlease be aware: Return type of the dataloader will not be a Tensor as expected butm;5;124m"2023-11-19T03:55:52.973Z] 101 124m"38;5;124m a MetaTensor instead! This is because 38;5;124m'38;5;124mspawnm;5;124m'38;5;124m creates a new process where TRACK_METAm;5;124m"2023-11-19T03:55:52.973Z] 102 124m"38;5;124m is initialized to True again. Context:TRACK_META is set to False andm;5;124m"2023-11-19T03:55:52.973Z] 103 124m"38;5;124m multiprocessing_context to spawnm;5;124m"2023-11-19T03:55:52.973Z] 104 )
[2023-11-19T03:55:52.973Z] --> 106 28;43msuper;49mm(43m)38;5;241;43m.m;5;21;43m__init;49mm(43mdatasetm;5;241;43m=mmdatasetmm,43m 43mnum_workersm;5;241;43m=mmnum_workersmm,43m 38;5;241;43mm;5;241;43mmmkwargsmm)2023-11-19T03:55:52.973Z]
[2023-11-19T03:55:52.973Z] File /usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:307, in DataLoader.__init__34m(self, dataset, batch_size, shuffle, sampler, batch_sampler, num_workers, collate_fn, pin_memory, drop_last, timeout, worker_init_fn, multiprocessing_context, generator, prefetch_factor, persistent_workers, pin_memory_device)023-11-19T03:55:52.973Z] 305 66;03m# We cannot check shuffle is not None here, since previously shuffle=False was the default.m
[2023-11-19T03:55:52.974Z] 306 28;01melif;00m shuffle 129;01mnot;00m 129;01min;00m {28;01mFalse;00m, 28;01mNone;00m}:
[2023-11-19T03:55:52.974Z] --> 307 28;01mraise;00m 167;01mValueError;00m(
[2023-11-19T03:55:52.974Z] 308 124m"38;5;124mDataLoader with IterableDataset: expected unspecified 38;5;124m"2023-11-19T03:55:52.974Z] 309 124m"38;5;124mshuffle option, but got shuffle=38;5;132;01m{}m;5;124m"38;5;241m.rmat(shuffle))
[2023-11-19T03:55:52.974Z] 311 28;01mif;00m sampler 129;01mis;00m 129;01mnot;00m 28;01mNone;00m:
[2023-11-19T03:55:52.974Z] 312 66;03m# See NOTE [ Custom Samplers and IterableDataset ]m
[2023-11-19T03:55:52.974Z] 313 28;01mraise;00m 167;01mValueError;00m(
[2023-11-19T03:55:52.974Z] 314 124m"38;5;124mDataLoader with IterableDataset: expected unspecified 38;5;124m"2023-11-19T03:55:52.974Z] 315 124m"38;5;124msampler option, but got sampler=38;5;132;01m{}m;5;124m"38;5;241m.rmat(sampler))
[2023-11-19T03:55:52.974Z]
[2023-11-19T03:55:52.974Z] ValueError: DataLoader with IterableDataset: expected unspecified shuffle option, but got shuffle=True
@KumoLiu KumoLiu added the bug Something isn't working label Nov 19, 2023
KumoLiu added a commit to KumoLiu/tutorials that referenced this issue Nov 20, 2023
@KumoLiu
fix Project-MONAI#1576
fd54836 
Signed-off-by: KumoLiu <yunl@nvidia.com>
@KumoLiu KumoLiu mentioned this issue Nov 20, 2023
Merged
5 tasks
ericspod pushed a commit that referenced this issue Nov 20, 2023
@KumoLiu
Fix shuffle error in 2d_slices_from_3d_sampling.ipynb (#1577)
c954aaa 
Fix #1576

The error was introduced by
Project-MONAI/MONAI#7180.

### Checks
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [ ] Avoid including large-size files in the PR.
- [ ] Clean up long text outputs from code cells in the notebook.
- [ ] For security purposes, please check the contents and remove any
sensitive info such as user names and private key.
- [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use
relative paths for tutorial repo files (3) put figure and graphs in the
`./figure` folder
- [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>`

Signed-off-by: KumoLiu <yunl@nvidia.com>
boneseva pushed a commit to boneseva/MONAI-tutorials that referenced this issue Apr 21, 2024
@KumoLiu
Fix shuffle error in 2d_slices_from_3d_sampling.ipynb (Project-MONA…
a4e80f0 
…I#1577)

Fix Project-MONAI#1576

The error was introduced by
Project-MONAI/MONAI#7180.

### Checks
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [ ] Avoid including large-size files in the PR.
- [ ] Clean up long text outputs from code cells in the notebook.
- [ ] For security purposes, please check the contents and remove any
sensitive info such as user names and private key.
- [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use
relative paths for tutorial repo files (3) put figure and graphs in the
`./figure` folder
- [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>`

Signed-off-by: KumoLiu <yunl@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix shuffle error in 2d_slices_from_3d_sampling.ipynb KumoLiu/tutorials
1 participant
@KumoLiu