Bootstrapping Krippendorff's Alpha

Question

I should like to calculate - by bootstrapping Krippendorff\'s Alpha outcomes - a 95% confidence interval for Krippendorff\'s Alpha coefficient of Raters Reliability using R

Answer 1

Unfortunately, the bootstrap solution given by josliber doesn't do what you think it does. The problem is that boot() expects data in an nXm matrix while kripp.alpha() expects data in an mXn matrix. The solution given will run, as shown, but the resampling being done is not by subjects, but by raters, so with 4 raters in the example data set we have a small number of possible samples, with the possibility that the resampled set will come from a single rater (hence the conf interval includes 1.0).

One solution is to keep your data in the nXm form that boot uses, and add a matrix transpose before giving it to kripp.alpha().

alpha.boot <- function(data,x) {
   d <- t(data[x,])
   kripp.alpha(d,method="nominal")$value
}