text-processing

I have two files: file1 has the format: field1;field2;field3;field4 (file1 is initially unsorted) file2 has the format: field1 (file2 is sorted) I run the 2 following commands: sort -t\; -k1 file1 -o file1 # to sort file 1 join -t\; -1 1 -2 1 -o 1.1 1.2 1.3 1.4 file1 file2 I get the following message: join: file1:27497: is not sorted: line_which_was_identified_as_out_of_order Why is this happening ? (I also tried to sort file1 taking into consideration the entire line not only the first filed of the line but with no success) sort -t\; -c file1 doesn't output anything. Around line 27497, the

I have read the text1.similar("monstrous") and text1.concordance("monstrous") from this . Where I couldn't get the satisfactory answer for the difference between text1.concordance('monstrous') and text1.similar('monstrous') of natural language processing toolkit in python. So would you please give the explanation with an example in detail? Using concordance(token) gives you the context surrounding the argument token . It will show you the sentences where token appears. Using similar(token) returns a list of words that appear in the same context as token . In this case the the context is just

Following command outputs following lines of text on console git log --pretty=format:"%h;%ai;%s" --shortstat ed6e0ab;2014-01-07 16:32:39 +0530;Foo 3 files changed, 14 insertions(+), 13 deletions(-) cdfbb10;2014-01-07 14:59:48 +0530;Bar 1 file changed, 21 insertions(+) 5fde3e1;2014-01-06 17:26:40 +0530;Merge Baz 772b277;2014-01-06 17:09:42 +0530;Qux 7 files changed, 72 insertions(+), 7 deletions(-) I'm interested in having above format to be displayed like this ed6e0ab;2014-01-07 16:32:39 +0530;Foo;3;14;13 cdfbb10;2014-01-07 14:59:48 +0530;Bar;1;21;0 5fde3e1;2014-01-06 17:26:40 +0530;Merge Baz

I need to calculate BLEU score for identifying whether two sentences are similar or not.I have read some articles which are mostly about BLEU score for Measuring Machine translation accuracy.But i'm in need of a BLEU score to find out similarity between sentences in a same language[English].(i.e)(Both the sentences are in English).Thanks in anticipation. Well, if you just want to calculate the BLEU score, it's straightforward. Treat one sentence as the reference translation and the other as the candidate translation. dmcer For sentence level comparisons, use smoothed BLEU The standard BLEU

I am working on prediction problem using a large textual dataset. I am implementing Bag of Words Model. What should be the best way to get the bag of words? Right now, I have tf-idf of the various words and the number of words is too large to use it for further assignments. If I use tf-idf criteria, what should be the tf-idf threshold for getting bag of words? Or should I use some other algorithms. I am using python. Using the collections.Counter class >>> import collections, re >>> texts = ['John likes to watch movies. Mary likes too.', 'John also likes to watch football games.'] >>>