blog/clustering-analysis-k-means-and-hierarchical-clustering-by-hand-and-in-r/

[utterances-bot]

The complete guide to clustering analysis: k-means and hierarchical clustering by hand and in R - Stats and R

Learn how to perform clustering analysis, namely k-means and hierarchical clustering, by hand and in R. See also how the different clustering algorithms work

https://statsandr.com/blog/clustering-analysis-k-means-and-hierarchical-clustering-by-hand-and-in-r/

[AntoineSoetewey]

Comment written by royr2 on February 15, 2020 04:29:53:

Thanks for this amazing post ! Very clear, structured and pedagogical in nature. (Especially how you have repeated the same set of steps again and again for better assimilation).

Thanks Antoine, much appreciated !

[AntoineSoetewey]

Comment written by royr2 on February 15, 2020 04:29:53:

Thanks for this amazing post ! Very clear, structured and pedagogical in nature. (Especially how you have repeated the same set of steps again and again for better assimilation).

Thanks Antoine, much appreciated !

Comment written by Antoine Soetewey on February 15, 2020 06:02:53:

You are welcome ! Glad you liked it.

[AntoineSoetewey]

Comment written by Kristina on February 18, 2020 20:22:34:

Hi Antoine, Thank you for this really complete article ! 

I just read an article describing a two-step clustering, using hierarchical clustering first, and then a non hierarchical clustering using the "cluster means derived from the hierarchical clustering as starting point". They don't explain more, and I would really like to do this.

Nevertheless I don't know how to do it, what object do I have to use form the hclust and how ? Any suggestions ?

[AntoineSoetewey]

Comment written by Kristina on February 18, 2020 20:22:34:

Hi Antoine, Thank you for this really complete article ! 

I just read an article describing a two-step clustering, using hierarchical clustering first, and then a non hierarchical clustering using the "cluster means derived from the hierarchical clustering as starting point". They don't explain more, and I would really like to do this.

Nevertheless I don't know how to do it, what object do I have to use form the hclust and how ? Any suggestions ?

Comment written by Antoine Soetewey on February 19, 2020 12:38:50:

Hi Kristina, thanks for your comment.

First, final clusters from a hierarchical clustering can be extracted thanks to cutree(hclust, k = 2) where hclust is the result of your clustering and k = 2 is the number of desired clusters.

Second, perhaps the {prcr} package may be what you are looking for: "hierarchical clustering is performed to determine the initial partition for the subsequent k-means clustering procedure".

Hope this helps!

[AntoineSoetewey]

Comment written by Salman Alk on August 05, 2020 02:41:47:

very nice.
Thank you

[AntoineSoetewey]

Comment written by Salman Alk on August 05, 2020 02:41:47:

very nice.
Thank you

Comment written by Antoine Soetewey on August 05, 2020 05:39:52:

Glad you like it Salman!

[AntoineSoetewey]

Comment written by kathroji saikrishna on September 26, 2020 06:09:24:

I am impressed by the information that you have on this blog. It shows how well you understand this subject.

[AntoineSoetewey]

Comment written by kathroji saikrishna on September 26, 2020 06:09:24:

I am impressed by the information that you have on this blog. It shows how well you understand this subject.

Comment written by Antoine Soetewey on September 26, 2020 06:18:54:

Thanks for your kind feedback. I always try to write articles as complete as possible.

[johnsonlab]

Hi Antoine - thanks for the really comprehensive dive into this topic. Here's my question: is there a statistical test that provides a relative measure of how distinct K means clusters are from one another? Answers to similar questions elsewhere seem to be suggesting that the production of the clusters is its own validation of the "distinctness" of the clusters, and no post-test would be informative for that reason. I can convince myself that this is correct, but want to be sure. Do you have any additional guidance? All the best, Josh

[AntoineSoetewey]

Hi Antoine - thanks for the really comprehensive dive into this topic. Here's my question: is there a statistical test that provides a relative measure of how distinct K means clusters are from one another? Answers to similar questions elsewhere seem to be suggesting that the production of the clusters is its own validation of the "distinctness" of the clusters, and no post-test would be informative for that reason. I can convince myself that this is correct, but want to be sure. Do you have any additional guidance? All the best, Josh

Dear Josh,

Thanks for this interesting question.

First of all, I am not aware of any statistical test that provides a relative measure of how distinct clusters are. The fact that clusters are constructed following the k-means algorithm makes them, by definition, as different/distinct as possible (since similar points are grouped together and distant points are separated into different clusters). That does not necessarily mean there is no statistical test, it's just that I don't know any.

However, here are a few points I'd like to mention and which may be of interest to you:

• Thanks to your comment, I added the silhouette plot in this section. The two plots mentioned in this section can help you to (visually at least) determine how distinct clusters are.
• You can also compute the euclidian distance between points and their center. If these distances are small, it indicates that points are close to their center and clusters are thus more likely to be distinct.
• If you really need a statistical test (and if you don't find any online or in textbooks), you can also compare the final groups via a Student's t-test in case of 2 clusters, or an ANOVA in case of 3 groups or more. These statistical tests will not tell you how distinct the clusters are, but they will tell you if the clusters are significantly different from each other or not.

Hope this helps.

Regards,
Antoine

[johnsonlab]

Antoine, this was super helpful, for me and also some local colleagues as we wrap up an analysis. Great site, 4/4 stars would comment again!

[AntoineSoetewey]

Antoine, this was super helpful, for me and also some local colleagues as we wrap up an analysis. Great site, 4/4 stars would comment again!

Thanks for your kind feedback!

[danoctavian14]

Hi Antoine!

Loved the post and found it extremely informative!

I have a couple of questions

1. You mention that the quality of partition (BSS/TSS*100%) allows us to “compared to the quality of other partitions (with the same number of clusters!)”. What do you mean by “other partitions”? As in containing other/fewer variables?

2. Is the hierarchical cluster analysis “done” with running one of the five linkage methods? How do we know what’s the quality of the clustering (something equivalent to BSS/TSS). My instinct was to run a k mean using the number of clusters from the hierarchical model, which leads me to the third question:

3. What’s the difference between the 4 approaches from the section “Optimal number of clusters” (k means section) and hierarchical clustering? My understanding, based on this post, is that both tell us the same thing: how many clusters we “ideally” have (unless we choose to predefine the number subjectively).

In our example, in NbClust()we limited the number of clusters to 5. If for example I set the maximum number of clusters to, say, 1000, wouldn’t this basically produce the same result as the hierarchical model? (I can imagine that there may be situations when have over 1000 clusters from the hierarchical model, but for most practical purposes I assume that 1000 is a proxy for infinity)

Thanks a lot for this post and for the rest! I discovered it today and it made my day!

[AntoineSoetewey]

Dear Dan,

Thanks for your feedback and questions.

1. By partition I mean "how observations are divided into groups". So a partition can be seen as a classification. Regarding the quality of partitions: each time you run a k-means clustering on the same dataset, you may have a different final partition because initial centers are randomly assigned (unless specified in the function), resulting in slightly different final clusters and thus slightly different levels of quality. Quality of a classification (via BSS/TSS) can be used to assess how well observations are divided into the different clusters (the higher the better). It's a bit like the coefficient of determination, R^2, in a linear regression. However, two partitions can only be compared through their quality if they have the same number of clusters. This is because, if done on the same data, quality increases with the number of clusters so a partition with more clusters will by definition have a higher quality (or at least not worse, see the end of this section for more information). Now if you want to compare quality between partitions done on different variables via BSS/TSS, you can but I would still make sure to compare partitions with the same number of clusters for the reason I mentioned above.
2. You can assess the quality of a hierarchical clustering via a silhouette plot and/or the Dunn index. See more in this article.
3. It is true that both tell us how many clusters we ideally have. Except that for k-means clustering if you don’t have a precise idea about the number of clusters, you can find it thanks to some criteria and then evaluate the goodness of fit (and adapt the number of clusters if quality is not satisfactory for instance), whereas for hierarchical clustering you usually find the number of clusters based on the dendrogram and then assess its quality based on some criteria (some of them which follows the same principle than for k-means).

I see your point of setting the maximum number of clusters to 1000 to remove constraints and see what the model gives. However, you will encounter an issue in R because the maximum number of clusters cannot be larger than the number of observations (because in theory the maximum number of clusters you can have is the number of observations, such that each class is a singleton). And even if you specify max.nc = nrow(dat) (with dat corresponding to the name of your dataset) in the NbClust() function, you will most likely encounter an issue too (saying that the system is computationally singular), so the maximum number of clusters should be strictly lower than the number of observations (and preferably even less than that). Second, it may gives the same result as the hierarchical model, but it will not always be the case because in the NbClust() function the method has to be specified (one of: "ward.D", "ward.D2", "single", "complete", "average", "mcquitty", "median", "centroid", "kmeans"). As the k-means and hierarchical clustering algorithms differ, it may therefore still gives you a different number of optimal clusters depending on the method used.

Hope this helps.

Regards,
Antoine

[peixinha]

how to characterize groups after a hierarchical cluster.

what I am struggling (and actually missing in all tutorials on clustering) is what's next.
How do I characterize my groups. What are the indicators/tests to be used for a clear interpretation of the groups.

thanks

Grazia

[AntoineSoetewey]

Thanks a lot for your question Grazia!

It is true that I did not mention what's next.

I have added a section at the end of the article to briefly explain what you could do after your clustering. See this section.

Hope it helps.

Regards,
Antoine

[yasout]

Thank you for writing your blog on how to carry out hierarchical clustering manually.
My question is: what do you do when you have two sets of groups with the same minimum? Which one do you select?

[AntoineSoetewey]

Hello,

It happens sometimes. In that case, choose any of the two. When I create exams and that happens, I consider both final solutions as the correct one.