How to fill NA with median?

Question

Example data:

set.seed(1)
df <- data.frame(years=sort(rep(2005:2010, 12)), 
                 months=1:12, 
                 value=c(rnorm(60),NA,NA,NA,NA,         


        

Answer 1

Or with ave

df <- data.frame(years=sort(rep(2005:2010, 12)),
months=1:12,
value=c(rnorm(60),NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA))
df$value[is.na(df$value)] <- with(df, ave(value, months, 
   FUN = function(x) median(x, na.rm = TRUE)))[is.na(df$value)]

---

Since there are so many answers let's see which is fastest.

plyr2 <- function(df){
  medDF <- ddply(df,.(months),summarize,median=median(value,na.rm=TRUE))
df$value[is.na(df$value)] <- medDF$median[match(df$months,medDF$months)][is.na(df$value)]
  df
}
library(plyr)
library(data.table)
DT <- data.table(df)
setkey(DT, months)


benchmark(ave = df$value[is.na(df$value)] <- 
  with(df, ave(value, months, 
               FUN = function(x) median(x, na.rm = TRUE)))[is.na(df$value)],
          tapply = df$value[61:72] <- 
            with(df, tapply(value, months, median, na.rm=TRUE)),
          sapply = df[61:72, 3] <- sapply(split(df[1:60, 3], df[1:60, 2]), median),
          plyr = ddply(df, .(months), transform, 
                       value=ifelse(is.na(value), median(value, na.rm=TRUE), value)),
          plyr2 = plyr2(df),
          data.table = DT[,value := ifelse(is.na(value), median(value, na.rm=TRUE), value), by=months],
          order = "elapsed")
        test replications elapsed relative user.self sys.self user.child sys.child
3     sapply          100   0.209 1.000000     0.196    0.000          0         0
1        ave          100   0.260 1.244019     0.244    0.000          0         0
6 data.table          100   0.271 1.296651     0.264    0.000          0         0
2     tapply          100   0.271 1.296651     0.256    0.000          0         0
5      plyr2          100   1.675 8.014354     1.612    0.004          0         0
4       plyr          100   2.075 9.928230     2.004    0.000          0         0

I would have bet that data.table was the fastest.

[ Matthew Dowle ] The task being timed here takes at most 0.02 seconds (2.075/100). data.table considers that insignificant. Try setting replications to 1 and increasing the data size, instead. Or timing the fastest of 3 runs is also a common rule of thumb. More verbose discussion in these links :

• Evidence that data.table isn't always fastest
• Benchmarks in Averaging column values for specific sections of data corresponding to other column values
• London R presentation, June 2012 (slide 21 headed "Other")
• A transform by group benchmark in an extreme case