information-retrieval

I am trying to calculate similarity. First of all i used RAKE library to extract the keywords from the crawled jobs. Then I put the keywords of every jobs into separate array and then combined all those arrays into documentArray. documentArray = ['Anger command,Assertiveness,Approachability,Adaptability,Authenticity,Aggressiveness,Analytical thinking,Molecular Biology,Molecular Biology,Molecular Biology,molecular biology,molecular biology,Master,English,Molecular Biology,,Islamabad,Islamabad District,Islamabad Capital Territory,Pakistan,,Rawalpindi,Rawalpindi,Punjab,Pakistan'"], ['competitive

Does Lucene provide a means to boost fresh documents? For example suppose that the Lucene document includes a date field. Is it possible, without having the user to alter her query anyhow, to present the most recent documents with a higher score? I do not want to resort to a coarse "sort by date" solution as it will completely cancel the scoring algorithm. Use Document.setBoost(float value) when putting documents into the index. You can either constantly re-adjust the value on existing documents, OR have a float value that increments with date, so that you only need to apply it to the time

I am looking for a simple script that can find frequencies of words for a given document (probably by using portable stemmer). Is there any library or simple script that does this process? use nltk import nltk YOUR_STRING = "Your words" words = [w for w in YOUR_STRING.split()] freq_dist = nltk.FreqDist(words) tokens = freq_dist.keys() #50 most frequent most_frequent = tokens[:50] #50 least frequent least_frequent = tokens[-50:] You should be able to count words. Use a collections.Counter or a dict , depending on what you need. That part is easy, but if it isn't you can find the answer by

I have question about how to evaluate the information retrieve result is good or not such as calculate the relevant document rank, recall, precision ,AP, MAP..... currently, the system is able to retrieve the document from the database once the users enter the query. The problem is I do not know how to do the evaluation. I got some public data set such as "Cranfield collection" dataset link it contains 1.document 2.query 3.relevance assesments DOCS QRYS SIZE* Cranfield 1,400 225 1.6 May I know how to use do the evaluation by using "Cranfield collection" to calculate the relevant document rank,

I am looking for a document search engine (like Xapian, Whoosh, Lucene, Solr, Sphinx or others) which is capable of searching partial terms. For example when searching for the term "brit" the search engine should return documents containing either "britney" or "britain" or in general any document containing a word matching r *brit* Tangentially, I noticed most engines use TF-IDF (Term frequency-Inverse document frequency) or its derivatives which are based on full terms and not partial terms. Are there any other techniques that have been successfully implemented besides TF-IDF for document

I am working on getting the performance parameters of a tcp connection and one these parameters is the bandwidth. I am intending to use the tcp_info structure supported from linux 2.6 onwards, which holds the meta data about a tcp connection. The information can be retrieved using the getsockopt() function call on tcp_info . I have spent lot of time finding a good documentation which explains all the parameters in that structure, but couldn't find one. Also I tested a small program to retrieve the values from tcp_info for a tcp connection where I found the measured MSS values for most of the