ValueError: Expected argument task to either be 'binary', 'multiclass' or 'multilabel' but got 1

[saadtariq-ds]

vad_task = VoiceActivityDetection(ami, duration=2., batch_size=128)
vad_model = PyanNet(task=vad_task, sincnet={'stride': 10})
trainer = pl.Trainer(gpus=self.gpus, max_epochs=self.max_epochs)
trainer.fit(vad_model)

[waoooNiuBility]

I've met the question as you too，How did you solve it

[kaleaniket]

I'm also getting the same error on Colab, Is there any fix to this?

[saadtariq-ds]

I solved by downgrading torchmetrics version. Current version torchmetrics==0.10.3

[kaleaniket]

I solved by downgrading torchmetrics version. Current version torchmetrics==0.10.3

Thanks. It worked for me as well.

[toto6038]

So it looks like there's a breaking change on torchmetrics' API (AUROC.__new__ in .../miniconda3/envs/pyannote/lib/python3.8/site-packages/torchmetrics/classification/auroc.py). Downgrading torchmetrics may serve as a workaround, but Pyannote has to adapt to the new API eventually.

[hbredin]

but Pyannote has to adapt to the new API eventually.

Yes, indeed. Feel free to contribute this "adaptation" via a pull request.

[toto6038]

but Pyannote has to adapt to the new API eventually.

Yes, indeed. Feel free to contribute this "adaptation" via a pull request.

I'll do it

[saadtariq-ds]

Thanks everyone

[hbredin]

Re-opening the issue until @toto6038's PR fixes it.
Thanks @toto6038 ;-)

[toto6038]

Edit

Related PR: #1181

TL;DR

A release of minor update is enough to avoid the problem as the solution has been done already.

Detail

After some investigation to the source code, it looks like the existing code has already mitigated the problem and therefore there will be no PR. To better explain the cause of problem, we first examine the version of Pyannote that users get from pip:

pyannote-audio/pyannote/audio/tasks/segmentation/mixins.py

Lines 126 to 132 in 7ead17e

	def default_metric(
	self,
	) -> Union[Metric, Sequence[Metric], Dict[str, Metric]]:
	"""Returns macro-average of the area under the ROC curve"""
	num_classes = len(self.specifications.classes)
	return AUROC(num_classes, pos_label=1, average="macro", compute_on_step=False)

It's the change of the interface of AUROC's __new__() that results in the ValueError issue.

The latest code in develop branch has done similar work as that in torchmetrics to return different object depending on the type of the task.

pyannote-audio/pyannote/audio/tasks/segmentation/mixins.py

Lines 143 to 152 in b56add2

	if self.specifications.problem == Problem.BINARY_CLASSIFICATION:
	return BinaryAUROC(compute_on_cpu=True)
	elif self.specifications.problem == Problem.MULTI_LABEL_CLASSIFICATION:
	return MultilabelAUROC(num_classes, average="macro", compute_on_cpu=True)
	elif self.specifications.problem == Problem.MONO_LABEL_CLASSIFICATION:
	return MulticlassAUROC(num_classes, average="macro", compute_on_cpu=True)
	else:
	raise RuntimeError(
	f"The {self.specifications.problem} problem type hasn't been given a default segmentation metric yet."
	)

[hbredin]

Oh! Thanks @toto6038 :-)

To anyone encountering the same issue, and until this fix has been released, please install pyannote.audio from develop branch (where the issue has been fixed already).