Replies: 15 comments 5 replies
-
|
Thanks for reporting this, I think the Have you tried or |
Beta Was this translation helpful? Give feedback.
-
|
@wyli Thanks for the quick reply! With spawn ( With Working results with |
Beta Was this translation helpful? Give feedback.
-
|
@wyli This topic is sadly not resolved however. Using ThreadDataloader modifies the logic of my real code, I believe the affine matrix for the spatial transforms is getting lost in the process. The transforms, similar to DeepEdit, look like this: # Initial transforms on the CPU which does not hurt since they are executed asynchronously and only once
InitLoggerd(args), # necessary if the dataloader runs in an extra thread / process
LoadImaged(keys=("image", "label"), reader="ITKReader"),
EnsureChannelFirstd(keys=("image", "label")),
NormalizeLabelsInDatasetd(keys="label", label_names=labels, device=device),
Orientationd(keys=["image", "label"], axcodes="RAS"),
Spacingd(keys=["image", "label"], pixdim=spacing),
CropForegroundd(keys=("image", "label"), source_key="image", select_fn=threshold_foreground),
ScaleIntensityRanged(keys="image", a_min=0, a_max=43, b_min=0.0, b_max=1.0, clip=True), # 0.05 and 99.95 percentiles of the spleen HUs
### Random Transforms ###
RandCropByPosNegLabeld(keys=("image", "label"), label_key="label", spatial_size=args.train_crop_size, pos=0.6, neg=0.4) if args.train_crop_size is not None else NoOpd(),
DivisiblePadd(keys=["image", "label"], k=64, value=0) if args.inferer == "SimpleInferer" else NoOpd(), # UNet needs this
RandFlipd(keys=("image", "label"), spatial_axis=[0], prob=0.10),
RandFlipd(keys=("image", "label"), spatial_axis=[1], prob=0.10),
RandFlipd(keys=("image", "label"), spatial_axis=[2], prob=0.10),
RandRotate90d(keys=("image", "label"), prob=0.10, max_k=3),
# Move to GPU
ToTensord(keys=("image", "label"), device=device, track_meta=False),With the Dataloader the resulting shape is torch.Size([1, 3, 224, 224, 320]) I have run into this error before and I believe this is due to |
Beta Was this translation helpful? Give feedback.
-
|
yes, please use |
Beta Was this translation helpful? Give feedback.
-
|
Does not change anything. Do I have to call that sooner? The ToTensord call is after all the relevant transforms imo. |
Beta Was this translation helpful? Give feedback.
-
|
Same problem, even with ToTensord right at the start: InitLoggerd(args), # necessary if the dataloader runs in an extra thread / process
LoadImaged(keys=("image", "label"), reader="ITKReader"),
ToTensord(keys=("image", "label"), device=device, track_meta=True),
EnsureChannelFirstd(keys=("image", "label")),
NormalizeLabelsInDatasetd(keys="label", label_names=labels, device=device),
Orientationd(keys=["image", "label"], axcodes="RAS"),
Spacingd(keys=["image", "label"], pixdim=spacing),
CropForegroundd(keys=("image", "label"), source_key="image", select_fn=threshold_foreground),
ScaleIntensityRanged(keys="image", a_min=0, a_max=43, b_min=0.0, b_max=1.0, clip=True), # 0.05 and 99.95 percentiles of the spleen HUs |
Beta Was this translation helpful? Give feedback.
-
|
I just changed my code to Dataset instead of PersistentDataset, just to be sure this is no caching effect. Same results on Dataset |
Beta Was this translation helpful? Give feedback.
-
|
ok... perhaps this |
Beta Was this translation helpful? Give feedback.
-
|
You don't need this transform ( This transform reassigns label indexes when the indexes are not originally sequential: https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/deepedit/transforms.py#L86-L120 |
Beta Was this translation helpful? Give feedback.
-
|
Same result, so the transforms still don't work as expected, plus the code then crashed later of course since information is missing. The code is extremely similar to this one https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/deepedit/transforms.py#L86 , so it should check if the tensor is normal or a MetaTensor |
Beta Was this translation helpful? Give feedback.
-
|
@wyli Would be great if you find the time to check that out 😊 As I said, with Dataloader it works, with ThreadDataloader it doesn't. Just for completeness I'll paste the calling code for both below. - train_loader = DataLoader(
- train_ds, shuffle=True, num_workers=args.num_workers, batch_size=1, multiprocessing_context='spawn', persistent_workers=True,
+ train_loader = ThreadDataLoader(
+ train_ds, shuffle=True, num_workers=args.num_workers, batch_size=1, multiprocessing_context='spawn', use_thread_workers=True#, persistent_workers=True, |
Beta Was this translation helpful? Give feedback.
-
|
sure, I forgot to mention that the issue of |
Beta Was this translation helpful? Give feedback.
-
|
Ah no worries there, I am using args.num_workers==1 per default. Good to know anyways, then I won't increase it. I found old code mentioning setting num_workers to 0 but that no longer works. |
Beta Was this translation helpful? Give feedback.
-
Hi @diazandr3s I'm not sure about the root cause of this deepedit transform + ThreadDataloader issue, please have a look if you have time.. thanks! (converting this to a discussion for now, please feel free to create a bug report if it's not a usage question) |
Beta Was this translation helpful? Give feedback.
-
|
Since changing one single line, DataLoader to ThreadDataLoader, changes how transforms work and renders ThreadDataLoader unusable to me I would consider this to be a bug. I will paste below how the different batchdata's look after the LoadImaged() call (done with PrintDatad(), a transform I added). Should be easy to debug for the person who created the code. The problem is that with use_thread_workers=True, Tensors are returned instead of MetaTensors, which means the Meta dicts get lost. I guess this is the dict used by the other transforms and they just ignore the other setting where this is information is: image_meta_dict/original_affine with use_thread_workers=True with use_thread_workers not set |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for reporting this @matt3o. Just for me to understand best the task here. Are you trying to segment single-label (binary), right? Is it for tumour segmentation using PETs? If the network output is a single output layer, what type of activation are you using after getting that value and then computing the CrossEntropyLoss? From the pre-transforms, I see some of them and some arguments are not typical:
Do you really want to only consider HU from 0 to 43?
My hypothesis is that the issue might come from applying all these transforms to your dataset :/ |
Beta Was this translation helpful? Give feedback.
-
|
Hey @diazandr3s! Thanks for the quick response. This code is running on the AutoPET dataset currently, so whole-body interactive segmentation. I took over the code from my tutor, so most of those settings have been done by him. As I said, the code works perfectly with DataLoader but not with ThreadDataLoader with use_thread_workers=True.
Since the code works with the normal dataloader I don't think this is related in any way to the transforms.. |
Beta Was this translation helpful? Give feedback.
-
|
@wyli I do have some more context for you now. The I now get: Not sure if it is intended that changing a single dataloader flag leads to a completely different returned data type. So the affine information is not getting read correctly because of the loaded image and label not being of type MetaTensor. Even by explicitly adding a Maybe that helps debugging the issue. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @matt3o I'm trying to replicate what you mentioned with import torch
import monai.transforms as mt
from monai.data import ArrayDataset, DataLoader, MetaTensor
NETWORK_INPUT_SHAPE = (1, 128, 128, 256)
NUM_IMAGES = 50
def get_xy():
xs = [256 * MetaTensor(torch.rand(NETWORK_INPUT_SHAPE)) for _ in range(NUM_IMAGES)]
ys = [MetaTensor(torch.rand(NETWORK_INPUT_SHAPE)) for _ in range(NUM_IMAGES)]
return xs, ys
transform = mt.Compose([mt.ToDevice(device="cpu")])
def get_data_loader():
x, y = get_xy()
dataset = ArrayDataset(x, seg=y, img_transform=transform, seg_transform=transform)
loader = DataLoader(dataset, num_workers=1, batch_size=1)
return loader
if __name__ == "__main__":
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
train_loader = get_data_loader()
for x in train_loader:
print(type(x[0]))but the outputs are metatensors. could you help revise the script to show the issue? (also tried the threadloader works fine) |
Beta Was this translation helpful? Give feedback.
-
import glob
import os
import logging
import monai.transforms as mt
import torch
from monai.data import ArrayDataset, DataLoader, MetaTensor
from monai.data.dataset import Dataset, PersistentDataset
from monai.data import set_track_meta
NETWORK_INPUT_SHAPE = (1, 128, 128, 256)
NUM_IMAGES = 1
logger = logging.getLogger("sw_interactive_segmentation")
if logger.hasHandlers():
logger.handlers.clear()
logger.propagate = False
logger.setLevel(logging.DEBUG)
stream_handler = logging.StreamHandler()
# (%(name)s)
formatter = logging.Formatter(
fmt="[%(asctime)s.%(msecs)03d][%(levelname)s] %(funcName)s - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
stream_handler.setFormatter(formatter)
stream_handler.setLevel(logging.DEBUG)
logger.addHandler(stream_handler)
if __name__ == "__main__":
location = "/cvhci/data/AutoPET/AutoPET"
all_images = sorted(glob.glob(os.path.join(location, "imagesTr", "*.nii.gz")))
all_labels = sorted(glob.glob(os.path.join(location, "labelsTr", "*.nii.gz")))
datalist = [
{"image": image_name, "label": label_name}
for image_name, label_name in zip(all_images, all_labels)
] # if image_name not in bad_images]
datalist = datalist[0:1]
device = "cuda"
transform = mt.Compose(
[
mt.LoadImaged(
keys="image",
reader="ITKReader",
image_only=False,
simple_keys=True,
),
]
)
# train_ds = Dataset(datalist, transform)
train_ds = Dataset(
datalist, transform#, cache_dir='/tmp/cache/'
)
train_ds2 = Dataset(
datalist, transform#, cache_dir='/tmp/cache/'
)
train_loader = DataLoader(
train_ds,
shuffle=True, num_workers=1, batch_size=1, multiprocessing_context='spawn', #persistent_workers=True,
)
train_loader2 = DataLoader(
train_ds2,
shuffle=True, num_workers=1, batch_size=1#, multiprocessing_context='spawn', #persistent_workers=True,
)
set_track_meta(False)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
for x in train_loader:
print(type(x["image"]))
for x in train_loader2:
print(type(x["image"]))
print(type(transform(datalist[0])["image"]))This code triggers the bug. Just load any image I think, I just copied over some AutoPET code. expected would be for the first one to return a torch.Tensor as well. I was not aware that the code I am using is setting Update: Pasted the code twice and undid that. Tested and it worked on my machine. I'll open another bug report soon if I don't get an answer here, pretty sure this is a bug.. |
Beta Was this translation helpful? Give feedback.
-
Describe the bug
I just tried to get some sample code for #6626 but ran into a warning I have seen many times before. The problem appears when the transform pushed code the GPU and the data is then handed over from the Dataloader Thread to the main Thread.
This is no hard bug but it is very annoying since it gets spammed a lot.
Temporary workaround which I found is to add "persistent_workers=True," to the DataLoader, then the warning gets only shown at the end of the program, sometimes never.
Warning message:
To Reproduce
Run this code, minimal sample:
Expected behavior
No Cuda Warnings
Environment
Verified on different environments.
Additional context
Adding an evaluator further complicates the warnings and a new warning is now shown:
The code for that:
Beta Was this translation helpful? Give feedback.
All reactions