Why does dplyr's filter drop NA values from a factor variable?

Question

When I use filter from the dplyr package to drop a level of a factor variable, filter also drops the NA values. Here's an example:

library(dplyr)
set.seed(919)
(dat <- data.frame(var1 = factor(sample(c(1:3, NA), size = 10, replace = T))))
#    var1
# 1  <NA>
# 2     3
# 3     3
# 4     1
# 5     1
# 6  <NA>
# 7     2
# 8     2
# 9  <NA>
# 10    1

filter(dat, var1 != 1)
#   var1
# 1    3
# 2    3
# 3    2
# 4    2

This does not seem ideal -- I only wanted to drop rows where var1 == 1.

It looks like this is occurring because any comparison with NA returns NA, which filter then drops. So, for example, filter(dat, !(var1 %in% 1)) produces the correct results. But is there a way to tell filter not to drop the NA values?

Answer 1

You could use this:

 filter(dat, var1 != 1 | is.na(var1))
  var1
1 <NA>
2    3
3    3
4 <NA>
5    2
6    2
7 <NA>

And it won't.

Also just for completion, dropping NAs is the intended behavior of filter as you can see from the following:

test_that("filter discards NA", {
  temp <- data.frame(
    i = 1:5,
    x = c(NA, 1L, 1L, 0L, 0L)
  )
  res <- filter(temp, x == 1)
  expect_equal(nrow(res), 2L)
})

This test above was taken from the tests for filter from github.

Answer 2

I often map identical with mapply...

(note: I believe because of changes in R 3.6.0, set.seed and sample end up with different test data)

library(dplyr, warn.conflicts = FALSE)
set.seed(919)
(dat <- data.frame(var1 = factor(sample(c(1:3, NA), size = 10, replace = T))))
#>    var1
#> 1     3
#> 2     1
#> 3  <NA>
#> 4     3
#> 5     1
#> 6     3
#> 7     2
#> 8     3
#> 9     2
#> 10    1

filter(dat, var1 != 1)
#>   var1
#> 1    3
#> 2    3
#> 3    3
#> 4    2
#> 5    3
#> 6    2

filter(dat, !mapply(identical, as.numeric(var1), 1))
#>   var1
#> 1    3
#> 2 <NA>
#> 3    3
#> 4    3
#> 5    2
#> 6    3
#> 7    2

it works for numerics and strings as well (probably more common use case)...

library(dplyr, warn.conflicts = FALSE)
set.seed(919)
(dat <- data.frame(var1 = sample(c(1:3, NA), size = 10, replace = T),
                   var2 = letters[sample(c(1:3, NA), size = 10, replace = T)],
                   stringsAsFactors = FALSE))
#>    var1 var2
#> 1     3 <NA>
#> 2     1    a
#> 3    NA    a
#> 4     3    b
#> 5     1    b
#> 6     3 <NA>
#> 7     2    a
#> 8     3    c
#> 9     2 <NA>
#> 10    1    b

filter(dat, !mapply(identical, var1, 1L))
#>   var1 var2
#> 1    3 <NA>
#> 2   NA    a
#> 3    3    b
#> 4    3 <NA>
#> 5    2    a
#> 6    3    c
#> 7    2 <NA>

filter(dat, !mapply(identical, var2, 'a'))
#>   var1 var2
#> 1    3 <NA>
#> 2    3    b
#> 3    1    b
#> 4    3 <NA>
#> 5    3    c
#> 6    2 <NA>
#> 7    1    b