text-processing

I have a longitudinal data set generated by a computer simulation that can be represented by the following tables ('var' are variables): time subject var1 var2 var3 t1 subjectA ... t2 subjectB ... and subject name subjectA nameA subjectB nameB However, the file generated writes a data file in a format similar to the following: time t1 description subjectA nameA var1 var2 var3 subjectB nameB var1 var2 var3 time t2 description subjectA nameA var1 var2 var3 subjectB nameB var1 var2 var3 ...(and so on) I have been using a (python) script to process this output data into a flat text file so that I

I have a Corpus of tweets from twitter. I clean this corpus (removeWords, tolower, delete URls) and finally also want to remove punctuation. Here is my code: tweetCorpus <- tm_map(tweetCorpus, removePunctuation, preserve_intra_word_dashes = TRUE) The problem now is, that by doing so I also loose the hashtag (#). Is there a way to remove punctuation with tm_map but remain the hashtag? You could adapt the existing removePunctuation to suit your needs. For example removeMostPunctuation<- function (x, preserve_intra_word_dashes = FALSE) { rmpunct <- function(x) { x <- gsub("#", "\002", x) x <-

I have a text file containing data like this: This is just text ------------------------------- Username: SOMETHI C: [Text] Account: DFAG Finish time: 1-JAN-2011 00:31:58.91 Process ID: 2028aaB Start time: 31-DEC-2010 20:27:15.30 This is just text ------------------------------- Username: SOMEGG C: [Text] Account: DFAG Finish time: 1-JAN-2011 00:31:58.91 Process ID: 20dd33DB Start time: 12-DEC-2010 20:27:15.30 This is just text ------------------------------- Username: SOMEYY C: [Text] Account: DFAG Finish time: 1-JAN-2011 00:31:58.91 Process ID: 202223DB Start time: 15-DEC-2010 20:27:15.30 Is

If I have text with standard paragraph formatting (a blank line followed by an indent) such as text 1 it's easy enough to extract the paragraphs using text.split("\n\n"). Text 1: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus sit amet sapien velit, ac sodales ante. Integer mattis eros non turpis interdum et auctor enim consectetur, etc. Praesent molestie suscipit bibendum. Donec justo purus, venenatis eget convallis sed, feugiat vitae velit,etc. But what if I have text with non-standard paragraph formatting such as text 2? No blank lines and variable leading whitespace. Text

I have the following list of words: name,id,3 I need to have it double quoted like this: "name,id,3" I have tried sed 's/.*/\"&\"/g' and got: "name,id,3 Which has only one double quote and is missing the closing double quote. I've also tried awk {print "\""$1"\""} with exactly the same result. I need help. Your input file has carriage returns at the end of the lines. You need to use dos2unix on the file to remove them. Or you can do this: sed 's/\(.*\)\r/"\1"/g' which will remove the carriage return and add the quotes. Use this to pipe your input into: sed 's/^/"/;s/$/"/' ^ is the anchor for

How can I use Python + NLTK to identify whether a sentence refers to the past/present/future ? Can I do this only using POS tagging? This seems a bit inaccurate, seems to me that I need to consider the sentence context and not only the words alone. Any suggestion for another library that can do that? It won't be too hard to do this yourself. This table should help you identify the different verb tenses and handling them will just be a matter of parsing the result of nltk.pos_tag(string) Im not sure if you want to get into all of the irregular verb tenses like 'could have been' etc... but if

I found the following code on the internet for calculating TFIDF: https://github.com/timtrueman/tf-idf/blob/master/tf-idf.py I added "1+" in the function def idf(word, documentList) so i won't get divided by 0 error: return math.log(len(documentList) / (1 + float(numDocsContaining(word,documentList)))) But i am confused for two things: I get negative values in some cases, is this correct? I am confused with line 62, 63 and 64. Code: documentNumber = 0 for word in documentList[documentNumber].split(None): words[word] = tfidf(word,documentList[documentNumber],documentList) Should TFIDF be