tm

I am using the 'tm' package in R to create a term document matrix using stemmed terms. The process is completing, but the resulting matrix includes terms that don't appear to have been stemmed, and I'm trying to understand why that is and how to fix it. Here is the script for the process, which uses a couple of online news stories as the sandbox: library(boilerpipeR) library(RCurl) library(tm) # Pull the relevant parts of the news stories using 'boilerpipeR' and 'RCurl' url <- "http://blogs.wsj.com/digits/2015/07/14/google-mozilla-disable-flash-over-security-concerns/" extract <-

I have corpus roughly with 75 MB data. I am trying to use the following command tm_map(doc.corpus, removeWords, stopwords("english")) tm_map(doc.corpus, tolower) This two alone functions are taking at least 40 mins to run. I am looking for speeding up the process as I am using tdm matrix for my model. I have tried commands like gc() and memory.limit(10000000) very frequently but I am not able to speed up my process speed. I have a system with 4GB RAM and running a local database to read the input data. Hoping for suggestions to speed up! Maybe you can give quanteda a try library(stringi)

I did a lot of research on this and I still can't find a solution to this. I have extracted data from a German Facebook group that looks like from_ID from_name message created_time 12334543 Max Muster Dies war auch eine sehr sch<U+00F6>ne Bucht 2016-01-08T19:00:54+0000 I understand that <U+00F6> stands for the German Umlat ö. There are many other examples of Unicode replacing German Umlaute or other language specifc signs (no matter which language). No matter if I want to do a sentiment analysis or just produce a wordcloud I sometimes have issues with this. In case of the sentiment an issue is

I have a list of documents to process, and for each record I want to attach some metadata to the document "member" inside the "corpus" data structure that tm, the R package, generates (from reading in text files). This for-loop works but it is very slow , Performance seems to degrade as a function f ~ 1/n_docs. for (i in seq(from= 1, to=length(corpus), by=1)){ if(opts$options$verbose == TRUE || i %% 50 == 0){ print(paste(i, " ", substr(corpus[[i]], 1, 140), sep = " ")) } DublinCore(corpus[[i]], "title") = csv[[i,10]] DublinCore(corpus[[i]], "Publisher" ) = csv[[i,16]] #institutions } This may