How to calculate the OOB of random forest?

问题

I am comparing some models to get a best model. Now, I want to get an OOB error of random forest model to compare it with the cross-validation errors of some others model. Can I do the comparison? If I can, how can I get the OOB error by R code?

回答1:

To get the OOB of a random forest model in R you can:

library(randomForest)

set.seed(1)
model <- randomForest(Species ~ ., data = iris)

OOB error is in:

model$err.rate[,1]

where the i-th element is the (OOB) error rate for all trees up to the i-th.

one can plot it and check if it is the same as the OOB in the plot method defined for rf models:

par(mfrow = c(2,1))
plot(model$err.rate[,1], type = "l")
plot(model)

OOB is useful for picking hyper parameters mtry and ntree and should correlate with k-fold CV but one should not use it to compare rf to different types of models tested by k-fold CV. OOB is great since it is almost free as opposed to k-fold CV which takes k times to run.

An easy way to run a k-fold CV in R is:

define the folds (replace the 5 with k (positive integer >1) to run k -fold CV:

folds <- sample(1:5, size = nrow(iris), replace = T) #5 fold CV

this approach will not give equally sized folds (especially for smaller data sets), this is usually not a big deal.

table(folds)
#output
 1  2  3  4  5 
30 28 28 33 31

to remedy this:

folds <- sample(rep(1:5, length.out = nrow(iris)), size = nrow(iris), replace = F)

table(folds)
#output
 1  2  3  4  5 
30 30 30 30 30

run through the folds training the model on each of the 4 folds and prediction on the 5th. Here I just return a list of data frames containing the predictions and real values, one can customize the call to return any statistic he desires.

CV_rf <- lapply(1:5, function(x){ #5 corresponds to the number of folds defined earlier
  model <- randomForest(Species ~ ., data = iris[folds != x,])
  preds <- predict(model,  iris[folds == x,], type="response")
  return(data.frame(preds, real = iris$Species[folds == x]))
  })

You can use the same code to get the performance of the ridge model.

convert list of data frames to a data frame:

CV_rf <- do.call(rbind, CV_rf)

check accuracy

caret::confusionMatrix(CV_rf$preds, CV_rf$real) 
#part of output:
Overall Statistics

               Accuracy : 0.9533         
                 95% CI : (0.9062, 0.981)
    No Information Rate : 0.3333         
    P-Value [Acc > NIR] : < 2.2e-16

so here the accuracy is 0.9533

while the OOB for the 500th (500 is fit by default in rf) tree was:

model$err.rate[500,1]
#OOB 
0.04666667

They are the same defeating my point completely, but for instance try to run 10 fold CV or 3 fold and you will see they are not the same.

another approach is to use caret or mlr libraries. I don't use mlr but caret is really good for tasks like this. Here is something to get you started with caret and rf. Additionally caret has excellent documentation. I can recommend it even if you do not plan to use the package.

来源：https://stackoverflow.com/questions/47960427/how-to-calculate-the-oob-of-random-forest

标签

random-forest