Carrot2 workbench not able to process large data

Question

I wanted to cluster my data-set using carrot2 workbench. I have an input xml file with 65536 documents. I am using Lingo clustering algorithm.

But when I start the process, the workbench returns the result within few seconds having all the documents in the "other topics" cluster.

I have checked the clustering with smaller data-sets and I am getting the results.

Answer 1

Carrot2 Lingo algorithm was designed for small data sets, up to a thousand or so of documents. For larger data sets, you may want to try STC, which scales better.

Regardless of the algorithm, Carrot2 processes all data in-memory, so it will not scale to millions of documents. In the latter case you may want to look at Apache Mahout, for example.