inverse document frequency

Question

The inverse document freqency is defined as follows:

IDF(term,document) = tf(term) * log(1 + n/df(term))

where tf(term) = 'frequency of term in document', n = 'number of documents', df(term) = 'number of docs containing term'.

Just curious about df(term) - do I only count a document ones even if it contains the term more than once?

Also is it easy to determine this stat with lucene(.net)? I am only starting to use the latter and use a relational db at the moment.

Thanks.

Christian

Answer 1

For using idf with Lucene, check the API for example here.

You are right about the docs being counted only once. The idea is to get a function with a lower bound in the log part. Like this:

If you are interested in the idf theory behind the scenes, you may peep at this paper.

HTH!

Answer 2

Of course you have to count the DF(term) once. therefore, you should group the words to get distinct words.

See my class IDF here