Extracting unique rows from a data table in R [duplicate]

Answer 1

As mentioned by Seth the data.table package has evolved and now proposes optimized functions for this.

To all the ones who don't want to get into the documentation, here is the fastest and most memory efficient way to do what you want :

uniqueN(a)

And if you only want to choose a subset of columns you could use the 'by' argument :

uniqueN(a,by = c('V1','V2'))

EDIT : As mentioned in the comments this will only gives the count of unique rows. To get the unique values, use unique instead :

unique(a)

And for a subset :

unique(a[c('V1',"V2")], by=c('V1','V2'))

Answer 2

Before data.table v1.9.8, the default behavior of unique.data.table method was to use the keys in order to determine the columns by which the unique combinations should be returned. If the key was NULL (the default), one would get the original data set back (as in OPs situation).

As of data.table 1.9.8+, unique.data.table method uses all columns by default which is consistent with the unique.data.frame in base R. To have it use the key columns, explicitly pass by = key(DT) into unique (replacing DT in the call to key with the name of the data.table).

Hence, old behavior would be something like

library(data.table) v1.9.7-
set.seed(123)
a <- as.data.frame(matrix(sample(2, 120, replace = TRUE), ncol = 3))
b <- data.table(a, key = names(a))
## key(b)
## [1] "V1" "V2" "V3"
dim(unique(b)) 
## [1] 8 3

While for data.table v1.9.8+, just

b <- data.table(a) 
dim(unique(b)) 
## [1] 8 3
## or dim(unique(b, by = key(b)) # in case you have keys you want to use them

Or without a copy

setDT(a)
dim(unique(a))
## [1] 8 3