Calculate TF-IDF of documents using HBase as the datasource

Question

I want to calculate the TF (Term Frequency) and the IDF (Inverse Document Frequency) of documents that are stored in HBase.

I also want to save the calculated TF in a HBase table, also save the calculated IDF in another HBase table.

Can you guide me through?

I have looked at BayesTfIdfDriver from Mahout 0.4 but I am not getting a head start.

Mahout 0.2? That's ancient, from 4 years ago. Use 0.6, or 0.7 which is about to be released. — Sean Owen
@SeanOwen It is 0.4 corrected it. Apache pulled out the HBase data store from 0.5. So I am using 0.4. — JHS
I deleted it :) The idea is that there doesn't really need to be direct HBase integration. Use Mahout with Hadoop, and Hadoop with HBase. That's the idea. — Sean Owen
@SeanOwen Can you just tell me how can i give input from HBase to Mahout using Hadoop. Please. — JHS
I don't know anything about this part of the code. The question is, how do you use HBase as input for Hadoop? That's a Hadoop question. — Sean Owen

David David · Accepted Answer · 2012-06-11T15:37:58

The outline of a solution is pretty straight forward:

do a word count over your hbase tables, storing both term frequency and document frequency for each word
in your reduce phase aggregate the term frequency and document frequency for each word
Given a count of your documents, scan through your aggregated results one more time and calculate the IDF based off of the document frequency.

The wikipedia page on TF-IDF is a good reference to remember the details of the formula: http://en.wikipedia.org/wiki/Tf*idf