How to specify data paths for training and testing? #990
-
|
Since I have very little data, I want to use cross-validation to validate my model, but I have encountered difficulties in the example of radiology, I don't know how to prepare what kind of json content and how to set the path to find My data path, I have tried to create apps, but I still can't find my file, what should I do? Thanks in advance! Environment: From Docker Hub (projectmonai/monailabel) 5 weeks ago. My Main.py: .....
studies = "../../HNC-datasets"
parser = argparse.ArgumentParser()
parser.add_argument("-s", "--studies", default=studies)
parser.add_argument("-m", "--model", default="segmentation_hnc")
parser.add_argument("-t", "--test", default="train", choices=("train", "infer"))
args = parser.parse_args()
app_dir = os.path.dirname(__file__)
studies = args.studies
conf = {
"models": args.model,
"preload": "true",
}
app = MyApp(app_dir, studies, conf)
# Infer
if args.test == "infer":
sample = app.next_sample(request={"strategy": "first"})
image_id = sample["id"]
image_path = sample["path"]
# Run on all devices
for device in device_list():
# res = app.infer(request={"model": args.model, "image": image_id, "device": device})
res = app.infer(
request={"model": "vertebra_pipeline", "image": image_id, "device": device, "slicer": False}
)
label = res["file"]
label_json = res["params"]
test_dir = os.path.join(args.studies, "test_labels")
os.makedirs(test_dir, exist_ok=True)
label_file = os.path.join(test_dir, image_id + file_ext(image_path))
shutil.move(label, label_file)
print(label_json)
print(f"++++ Image File: {image_path}")
print(f"++++ Label File: {label_file}")
return
# Train
app.train(
request={
"model": args.model,
"max_epochs": 10,
"dataset": "CacheDataset", # PersistentDataset, CacheDataset
"train_batch_size": 1,
"val_batch_size": 1,
"multi_gpu": False,
"val_split": 0.1,
"train_ds": "cvDataSets/trainCV1.json",
"val_ds": "cvDataSets/testCV1.json"
},
)
.....self-created json file: {
"images_dir": "images",
"labels_dir": "labels",
"objects": {
"image_9": {
"image": {
"ext": ".nii.gz",
"info": {
"name": "image_9.nii.gz"
}
},
"labels": {
"ext": ".nii.gz",
"info": {
"name": "label_9.nii.gz"
}
}
},
....Run Logs: |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 5 replies
-
|
Not this json.. it's simple array of dict [{"image": "x1.nii.go", "label":"l1.nii.vz"}, {...}] There is sample code to refer.. I guess I gave you the link above.. There are two ways.. to pass.. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Hi @Minxiangliu, As @SachidanandAlle suggested, you just need to pass a list of dict. No need to modify the self-created JSON file. Here is an example of a user-defined validation set: https://github.com/Project-MONAI/MONAILabel/blob/testsMICCAI/sample-apps/deepedit_multilabel/lib/train.py#L242-L256 - you should define a similar method in the training file of your app. Regarding this error:
It is because there are no training samples. I hope this helps, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @diazandr3s , def partition_datalist(self, context: Context, shuffle=False):
with open(r'/HNC/HNC-DataSets/cvDataSets/trainCV1.json', 'r') as file:
train_ds = json.load(file)
with open(r'/HNC/HNC-DataSets/cvDataSets/testCV1.json', 'r') as file:
val_ds = json.load(file)
return train_ds, val_dstestCV1.json [
{
"image": "/HNC/HNC-DataSets/images/image_2.nii.gz",
"label": "/HNC/HNC-DataSets/labels/label_2.nii.gz"
},
{
"image": "/HNC/HNC-DataSets/images/image_6.nii.gz",
"label": "/HNC/HNC-DataSets/labels/label_6.nii.gz"
}
]error log: transforms: self._transforms = [
LoadImaged(keys=['image','label']),
EnsureChannelFirstd(keys=['image','label']),
Orientationd(keys=['image','label'], axcodes='RAS'),
Spacingd(keys=['image','label'], pixdim=self.target_spacing, mode=('bilinear', 'nearest')),
ScaleIntensityRanged(keys=['image'], a_min=0, a_max=30000,
b_min=0.0, b_max=1.0, clip=True),
CropForegroundd(keys=['image','label'],
source_key='image',
select_fn=lambda x:x>0.05,
margin=0,
return_coords=False),
SpatialPadd(keys=['image','label'], spatial_size=self.padd_size)
] |
Beta Was this translation helpful? Give feedback.
-
|
Just try this on python console... (if needed please take care of any compile errors for the snippet) import monai.transforms.*
d = [{"image": "/HNC/HNC-DataSets/images/image_2.nii.gz", "label": "/HNC/HNC-DataSets/labels/label_2.nii.gz"}]
d = LoadImaged(keys=['image','label'])(d)
print(f"{d['image'].shape} => {d['label'].shape}")
d = EnsureChannelFirstd(keys=['image','label'])(d)
print(f"{d['image'].shape} => {d['label'].shape}")
d = Orientationd(keys=['image','label'], axcodes='RAS')(d)
print(f"{d['image'].shape} => {d['label'].shape}") |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Minxiangliu, Thanks for the update. You did well creating both training and validation sets. From my experience, it seems either the volume wasn't loaded correctly or the file is corrupted. You'd need to find out which one. Hope this helps, |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for your kind help. I am sorry for the delay in updating the content. I spent a little time to make sure that my data is correct. Indeed, there are some errors in processing the data. After troubleshooting, I can train on the dataset with the specified split, thank you very much for your help! |
Beta Was this translation helpful? Give feedback.
-
|
Glad to hear this. Please select an answer to this discussion so others can benefit from this conversation as well :) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @diazandr3s , Sorry I want to ask a question, I refer to the data structure here to develop the model training, but I have difficulty when I want to start the service, how can I correctly command? Data structure: Run command: Results: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Minxiangliu, Let's not mix code versions. The one you're referring to is an "old" MONAI Label App version and doesn't work with the current version. The only bit that I took from that was the method to create training and validation partition lists: https://github.com/Project-MONAI/MONAILabel/blob/testsMICCAI/sample-apps/deepedit_multilabel/lib/train.py#L242-L256 Additionally, the dataset can be in any folder. You just need to pass the path to the MONAI Label server when starting. Hope this helps, |
Beta Was this translation helpful? Give feedback.
Hi @Minxiangliu,
As @SachidanandAlle suggested, you just need to pass a list of dict. No need to modify the self-created JSON file.
Here is an example of a user-defined validation set: https://github.com/Project-MONAI/MONAILabel/blob/testsMICCAI/sample-apps/deepedit_multilabel/lib/train.py#L242-L256 - you should define a similar method in the training file of your app.
Regarding this error:
It is because there are no training samples.
I hope this helps,