Clustering metrics

[SkafteNicki]

🚀 Feature

Lets add clustering metrics to TM:

• Dunn index -> New Metric: Dunn Index #2049
• Silhouette @stancld
• Rand score -> New metric: Rand Score  #2025
• Adjusted_mutual_info_score -> New metric: Adjusted mutual info score #2058
• Adjusted rand score -> New metric: Adjusted Rand Score #2032
• Calinski Harabasz score -> New metric: Calinski Harabasz Score #2036
• Completeness score -> New metrics: Homogeneity, Completness, V-Measure #2053
• Fowlkes mallows score -> New metric: Fowlkes-Mallows Index #2066
• Homogeneity completeness measure -> New metrics: Homogeneity, Completness, V-Measure #2053
• Mutual info score -> Mutual Information Score #2008
• Normalized mutual info score -> New metric: Normalized Mutual Information Score #2029
• Davies–Bouldin index -> New metric: Davies bouldin score #2071
• V measure score -> New metrics: Homogeneity, Completness, V-Measure #2053

Motivation

In Supervised Learning, the labels are known and evaluation can be done by calculating the degree of correctness by comparing the predicted values against the labels. However, in Unsupervised Learning, the labels are not known, which makes it hard to evaluate the degree of correctness as there is no ground truth.

That being said, it is still consistent that a good clustering algorithm has clusters that have small within-cluster variance (data points in a cluster are similar to each other) and large between-cluster variance (clusters are dissimilar to other clusters).

ref: https://towardsdatascience.com/7-evaluation-metrics-for-clustering-algorithms-bdc537ff54d2

CTA

pls also check our contribution guide: https://torchmetrics.readthedocs.io/en/stable/generated/CONTRIBUTING.html

[github-actions]

Hi! thanks for your contribution!, great first issue!

[matsumotosan]

Hi @SkafteNicki, I'd like to give Mutual Info Score a shot!

Should the implementation be done in a new directory like torchmetrics/clustering and torchmetrics/functional/clustering?

[SkafteNicki]

@matsumotosan cool thanks for wanting to contribute :]
Yes, please add new directories as I consider this a new domain of metrics.

[SkafteNicki]

Also if it help, I was fooling around with some code in the weekend, trying to calculate the contingency matrix and pair matrix which seems to be two common objects used for calculating clustering metrics:

p = torch.tensor([0, 0, 1, 2])
t = torch.tensor([0, 0, 1, 1])

p_u = len(torch.unique(p))
t_u = len(torch.unique(t))

lin = p + p_u * t
contingency_matrix = torch.bincount(t + t_u * p, minlength=p_u * t_u).reshape(p_u, t_u)

n_c = contingency_matrix.sum(dim=1)
n_k = contingency_matrix.sum(dim=0)
sum_squared = (contingency_matrix ** 2).sum()

pair_matrix = torch.zeros(2,2)
pair_matrix[1,1] = sum_squared - len(p)
pair_matrix[0,1] = (contingency_matrix * n_k).sum() - sum_squared
pair_matrix[1,0] = (contingency_matrix.T * n_c).sum() - sum_squared
pair_matrix[0,0] = len(p) ** 2 - pair_matrix[0,1] - pair_matrix[1,0] - sum_squared

probably needs to be improved but maybe useful.

[matsumotosan]

I tried calculating the contingency matrix based on sklearn's implementation. I'll compare it to the method you suggested since it does seem to be a key computation.

[matsumotosan]

I'd like to work on normalized mutual info and Dunn index next!

[SkafteNicki]

I'd like to work on normalized mutual info and Dunn index next!

Assigned :]

[matsumotosan]

@SkafteNicki Would it be useful if we accumulated all the inputs for clustering-related tests in a separate file similar to inputs.py for classification? I saw that your rand_score tests used the same test cases that I used for mutual_info and normalized_mutual_info. I could also add cases for Dunn index that may be useful for other metrics in the future.

[stancld]

@SkafteNicki I'll take Silhouette

[SkafteNicki]

@SkafteNicki Would it be useful if we accumulated all the inputs for clustering-related tests in a separate file similar to inputs.py for classification? I saw that your rand_score tests used the same test cases that I used for mutual_info and normalized_mutual_info. I could also add cases for Dunn index that may be useful for other metrics in the future.

@matsumotosan I think that is a good idea, anything to help reduce replicated code is always good.
Can you send a PR with this ? Else I do it

[matsumotosan]

I'll send a draft PR in a bit.

[matsumotosan]

@SkafteNicki I'll take adjusted mutual info score as well.

[m0saan]

Hello @SkafteNicki, which of these are not yet claimed?

[SkafteNicki]

Hello @SkafteNicki, which of these are not yet claimed?

Hi @m0saan,
Currently Fowlkes mallows score and Davies–Bouldin index is not yet claimed.

[matsumotosan]

@SkafteNicki I can take Fowlkes-Mallows.

[Borda]

Most have been done, Thank you all for your contributions! 🐿️