How to Calculate cosine similarity with tf-idf using Lucene and Java

Question

I have a query and a set of documents. I need to rank these documents based on the cosine similarity with tf-idf. Can someone please tell me what support I can get from Lucene to compute this ? What parameters I can directly calculate from Lucene (can I get tf, idf directly through some method in lucene?) and how to compute cosine similarity with Lucene (is there any function which directly returns cosine similarity if I pass two vectors of the query and the document ?)

Thanx in advance

Answer 1

Lucene already uses a pimped version of cosine similarity, so if you need the raw CS itself, it's probably doable. I recommend the official page that discusses Lucene scoring.

If you want to extract that info on your own, this would be an outline of the steps for tf:

1. index the corpus;
2. open an IndexReader;
3. iterate over all doc ids, 0 to maxDoc();
4. getTermFreqVector(doc, fieldName);
5. iterate over the parallel arrays tfv.getTerms() and tfv.getTermFrequencies().

As for the docFreq, use IndexReader.terms() and iterate over this calling termEnum.docFreq().