The situation is as follows:
I have a dataset of documents which I've manually assigned to a (ground) cluster based on their subject. I've then used Hierarchical Agglomerative Clustering (HAC) to automatically cluster that same dataset. I'm now trying to evaluate the HAC clusters using the pair counting f-measure (as described in Characterization and evaluation of similarity measures for pairs of clusterings by Darius Pfitzner, Richard Leibbrandt & David Powers).
The problem I'm facing however is that my manual clustering produced flat clusters (so no relation between the clusters what so ever), while the clusters found by HAC are hierarchical. So when looking at the dendrogram, based on the depth (horizontal line) you chose you have a different number of clusters (At depth 0 (the root node) you only have 1 cluster; at MAX depth, your number of clusters equals the number of elements in your dataset).
So, my questions now are:
- Do I need to select a depth (so that I have a fixed set of clusters) in order to use the pair counting f-measure (or am I missing something?).
- If so, what criteria do I use to determine this depth?