Text clustering using Scipy Hierarchy Clustering in Python

Question

I have a text corpus that contains 1000+ articles each in a separate line. I am trying to use Hierarchy Clustering using Scipy in python to produce clusters of related articles. This is the code I used to do the clustering

# Agglomerative Clustering
import matplotlib.pyplot as plt
import scipy.cluster.hierarchy as hac
tree = hac.linkage(X.toarray(), method="complete",metric="euclidean")
plt.clf()
hac.dendrogram(tree)
plt.show() 

and I got this plot

Then I cut off the tree at the third level with fcluster()

from scipy.cluster.hierarchy import fcluster
clustering = fcluster(tree,3,'maxclust')
print(clustering)

and I got this output: [2 2 2 ..., 2 2 2]

My question is how can I find the top 10 frequent words in each cluster in order to suggest a topic for each cluster?

Answer 1

You can do the following:

1. Align your results (your clustering variable) with your input (the 1000+ articles).
2. Using pandas library, you can use a groupby function with the cluster # as its key.
3. Per group (using the get_group function), fill up a defaultdict of integers for every word you encounter.
4. You can now sort the dictionary of word counts in descending order and get your desired number of most frequent words.

Good luck with what you're doing and please do accept my answer if it's what you're looking for.