term-document-matrix

I have been breaking my head over this one over the last few days. I searched all the SO archives and tried the suggested solutions but just can't seem to get this to work. I have sets of txt documents in folders such as 2000 06, 1995 -99 etc, and want to run some basic text mining operations such as creating document term matrix and term document matrix and doing some operations based co-locations of words. My script works on a smaller corpus, however, when I try it with the bigger corpus, it fails me. I have pasted in the code for one such folder operation. library(tm) # Framework for text

I gather Text documents (in Node.js) where one document i is represented as a list of words. What is an efficient way to compute the similarity between these documents, taking into account that new documents are coming as a sort of stream of documents? I currently use cos-similarity on the Normalized Frequency of the words within each document. I don't use the TF-IDF (Term frequency, Inverse document frequency) because of the scalability issue since I get more and more documents. Initially My first version was to start with the currently available documents, compute a big Term-Document matrix

I have a termDocumentMatrix created using the tm package in R. I'm trying to create a matrix/dataframe that has the 50 most frequently occurring terms. When I try to convert to a matrix I get this error: > ap.m <- as.matrix(mydata.dtm) Error: cannot allocate vector of size 2.0 Gb So I tried converting to sparse matrices using Matrix package: > A <- as(mydata.dtm, "sparseMatrix") Error in as(from, "CsparseMatrix") : no method or default for coercing "TermDocumentMatrix" to "CsparseMatrix" > B <- Matrix(mydata.dtm, sparse = TRUE) Error in asMethod(object) : invalid class 'NA' to dup_mMatrix_as

Purpose: I want to create a term-document matrix using a dictionary which has compound words, or bigrams , as some of the keywords . Web Search: Being new to text-mining and the tm package in R , I went to the web to figure out how to do this. Below are some relevant links that I found: FAQS on the tm-package website finding 2 & 3 word phrases using r tm package counter ngram with tm package in r findassocs for multiple terms in r Background: Of these, I preferred the solution that uses NGramTokenizer in the RWeka package in R , but I ran into a problem . In the example code below, I create

I have two sets of data: a set of tags (single words like php , html , etc) a set of texts I wish now to build a Term-Document-Matrix representing the number occurrences of the tags element in the text element. I have looked into R library tm , and the TermDocumentMatrix function, but I do not see the possibility to specify the tags as input. Is there a way to do that? I am open to any tool (R, Python, other), although using R would be great. Let's set the data as: TagSet <- data.frame(c("c","java","php","javascript","android")) colnames(TagSet)[1] <- "tag" TextSet <- data.frame(c("How to

how to store the sparsity and maximum term length of Term Document Matrix in separate variable in R while finding ngrams ? library(tm) library(RWeka) #stdout <- vector('character') #con <- textConnection('stdout','wr',local = TRUE) #reading the csv file worklog <- read.csv("To_Kamal_WorkLogs.csv"); #removing the unwanted columns cols <- c("A","B","C","D","E","F"); colnames(worklog)<-cols; worklog2 <- worklog[c("F")] #removing non-ASCII characters z=iconv(worklog2, "latin1", "ASCII", sub="") #cleaning the data Removing Date and Time worklog2$F=gsub("[0-9]+/[0-9]+/[0-9]+ [0-9]+:[0-9]+:[0-9]+ [A

I have a huge corpus, and I'm interested in only appearance of a handful of terms that I know up front. Is there a way to create a term document matrix from the corpus using the tm package, where only terms I specify up front are to be used and included? I know I can subset the resultant TermDocumentMatrix of the corpus, but I want to avoid building the full term document matrix to start with, due to memory size constraint. eipi10 You can modify a corpus to keep only the terms you want by building a custom transformation function. See the Vignette for the tm package and the help for the