Monai label validation split constant or not?

[vedicsharma]

Hello,
I wanted to ask that when Monai Label is setup in the required fashion - with a directory with the scans and then a directory inside that with the labels that gets populated - and we select a validation split, does the software automatically make a constant validation set that ensures no data leakage affecting the validation accuracy? I want to make sure this is the case, as it is important for me while interpreting the training and test scores I receive. Thank you!

[diazandr3s]

Hi @vedicsharma,

You can set up your validation set by redefining this method:

MONAILabel/monailabel/tasks/train/basic_train.py

Lines 398 to 416 in fc9b567

	def partition_datalist(self, context: Context, shuffle=False):
	# user can external validation/training datalist in the request
	val_datalist = self._load_external_ds(context.request.get("val_ds"))
	train_datalist = self._load_external_ds(context.request.get("train_ds", context.datalist))
	if not val_datalist:
	val_split = context.request.get("val_split", 0.0)
	if val_split > 0.0:
	train_datalist, val_datalist = partition_dataset(
	train_datalist, ratios=[(1 - val_split), val_split], shuffle=shuffle
	)
	else:
	train_datalist = context.datalist
	val_datalist = []
	if context.local_rank == 0:
	logger.info(f"Total Records for Training: {len(train_datalist)}")
	logger.info(f"Total Records for Validation: {len(val_datalist)}")
	return train_datalist, val_datalist

in the training class (i.e. segmentation, deepedit)

Currently, MONAI Label uses by default 20% of the training set for "validation":

MONAILabel/monailabel/tasks/train/basic_train.py

Line 163 in fc9b567

"val_split": 0.2,

If you want to avoid data leakage, redefining this method should be done.

You may find this useful: #990

Hope this helps,

[vedicsharma]

Thank you for your answer. What we did in our project was instead of using a separate validation set, to use the default method in MONAI label, which was to have a directory with the scans and then a directory inside that with the labels that gets populated. From what I can understand from the code, this means that MONAI label split it into an 80-20 split on its own and handled everything from there. From there we did the active learning, having MONAI select scans based on epistemic uncertainty, manually labelling them, and then submitting to further train the DynUnet model.

Was our approach still valid and expected to give proper results?

[diazandr3s]

Thanks for the update!

Was our approach still valid and expected to give proper results?

It should be god to give you an idea of how the model is being trained. Depending on the use case, you may want to consider validating the model on an texernal set.

[vedicsharma]

Hello, apologies for the late reply, having some more questions.
When starting our monai label server, we did this:

apptainer exec --nv --home / -B /home/monailabel:/workspace /apps/nvidia/containers/monai/monailabel.0.6.0/0.6.0 monailabel start_server --app /workspace/apps/radiology --studies /workspace/datasets/pv1Vibes --conf models deepedit --conf skip_scoring false --conf skip_strategies false --conf epistemic_enabled true --conf use_pretrained_model false > log_client.log 2>&1

Based on this, we are using monai label's active learning. However, I am unsure of what code exactly is being run; does basic_train.py, which is what you attached above actually get run, or is it something else? The reason I am asking: I trained the model with active learning, the way I explained in the previous comment, and I want to compare its performance on validation accuracy with that of a traditional ML model. However, I want to ensure that my validation accuracy is accurate here; this means no leaks, etc.

All the monai label tutorials I have seen online and in documentation outline doing it the way we have done it, with a single directory with all the scans, and monai label handles the validaiton on its own, rather than the creation of an external set. We did start from scratch, using completely unlabeled data in the beginning, then retrained our model after every 5 scans added. The only concern I have is whether data that was used to train the model could end up in the validation set.

I assume this is not the case, correct? Because the split is done randomly at the beginning, and we are not adding anymore data to the dataset, just annotating the data and then retraining the model after annotating. But I wanted to be sure. Thank you!

[diazandr3s]

Hi @vedicsharma,

That's right, the split is done randomly when you trigger training. However, if you trigger the training again, it doesn't consider the previous split. So it could happen that a portion of the previous validation set end up being part of the training set.

To make sure this doesn't happen, I suggest you separate the validation set as I tried to explain here: #1718 (comment)

Hope this helps,

[vedicsharma]

One more quick question, but thank you for your advice, this is what I ended up doing; I used an external dataset with similar code to here.

But one more quick question: I also want to run it as traditional machine learning, just to compare with active learning. To do this, what I am doing is when I start a new session (I move the the old model at add/radiology/model to an archive folder so the model folder is empty and so the model is untrained when i start) and then for directories, I leave the labels and image files in their folders, and then when i start it on 3d slicer I don't add anymore images/annotations, just click train and thus train on what is already there. This will do what i want, ie train it essentially as traditional machine learning rather than active learning, correct? Thank you!
.