Using DocumentTermMatrix on a Vector of First and Last Names

问题

I have a column in my data frame (df) as follows:

> people = df$people
> people[1:3]
[1] "Christian Slater, Tara Reid, Stephen Dorff, Frank C. Turner"     
[2] "Ice Cube, Nia Long, Aleisha Allen, Philip Bolden"                
[3] "John Travolta, Uma Thurman, Vince Vaughn, Cedric the Entertainer"

The column has 4k+ unique first/last/nick names as a list of full names on each row as shown above. I would like to create a DocumentTermMatrix for this column where full name matches are found and only the names that occur the most are used as columns. I have tried the following code:

> people_list = strsplit(people, ", ")

> corp = Corpus(VectorSource(people_list))

> dtm = DocumentTermMatrix(corp, people_dict)

where people_dict is a list of the most commonly occurring people (~150 full names of people) from people_list as follows:

> people_dict[1:3]
[[1]]
[1] "Christian Slater"

[[2]]
[1] "Tara Reid"

[[3]]
[1] "Stephen Dorff"

However, the DocumentTermMatrix function seems to not be using the people_dict at all because I have way more columns than in my people_dict. Also, I think that the DocumentTermMatrix function is splitting each name string into multiple strings. For example, "Danny Devito" becomes a column for "Danny" and "Devito".

> inspect(actors_dtm[1:5,1:10])
<<DocumentTermMatrix (documents: 5, terms: 10)>>
Non-/sparse entries: 0/50
Sparsity           : 100%
Maximal term length: 9
Weighting          : term frequency (tf)

    Terms
Docs 'g. 'jojo' 'ole' 'piolin' 'rampage' 'spank' 'stevvi' a.d. a.j. aaliyah
   1   0      0     0        0         0       0        0    0    0       0
   2   0      0     0        0         0       0        0    0    0       0
   3   0      0     0        0         0       0        0    0    0       0
   4   0      0     0        0         0       0        0    0    0       0
   5   0      0     0        0         0       0        0    0    0       0

I have read through all the TM documentation that I can find, and I have spent hours searching on stackoverflow for a solution. Please help!

回答1:

The default tokenizer splits text into individual words. You need to provide a custom function

commasplit_tokenizer <- function(x)
unlist(strsplit(as.character(x), ", "))

Note that you do not separate the actors before creating the corpus.

people <- character(3)
people[1] <- "Christian Slater, Tara Reid, Stephen Dorff, Frank C. Turner"     
people[2] <- "Ice Cube, Nia Long, Aleisha Allen, Philip Bolden"                
people[3] <- "John Travolta, Uma Thurman, Vince Vaughn, Cedric the Entertainer"

people_dict <- c("Stephen Dorff", "Nia Long", "Uma Thurman")

The control options didn't work with just Coprus, I used VCorpus

corp = VCorpus(VectorSource(people))
dtm = DocumentTermMatrix(corp, control = list(tokenize = 
commasplit_tokenizer, dictionary = people_dict, tolower = FALSE))

All of the options are passed within control, including:

1. tokenize - function
2. dictionary
3. tolower = FALSE

Results:

as.matrix(dtm)
Terms
Docs Nia LOng Stephen Dorff Uma Thurman
   1        0             1           0
   2        0             0           0
   3        0             0           1

I hope this helps

来源：https://stackoverflow.com/questions/43441713/using-documenttermmatrix-on-a-vector-of-first-and-last-names