Clustering using LLM's

[kitsamho]

It would be super cool to see some form of sklearn clustering / topic modelling implementation where the inputs are texts and outputs are clusters/topics in the data.

[OKUA1]

@kitsamho, I think this is a very interesting idea. Could you elaborate a bit more on how you envision it to work?

Just doing the topic extraction does not seem to be too complicated, but in this case the model might produce similar, but not identical topics which in extreme cases might result in n_clusters = n_samples.

So maybe the n_clusters could be a hyperparameter and the algorithm works in 2 steps:

• Topic extraction -> produces a topic per sample
• Topic clustering -> groups topics by their similarity into n clusters and produces a "central" topic for each cluster

Of course an alternative could be to pack all samples into a single prompt and specify the constraint on the number of topics right away, but this will only work for relatively small datasets.

What do you think ? Or maybe you have a specific dataset in mind on which you could demonstrate the desired result ?

[kitsamho]

@OKUA1 let me think on this!

[jorgeston]

Maybe a good idea is using GPTSummarizer model with max_words=3 (or 2, 1) in order to get "topics", .i.e., a specific word o phrase which summarizes the idea of text in few words.