How to compute confidence intervall for Krippendorf's Alpha in R?

Question

I am sure this is realted to Bootstrapping Krippendorff's Alpha. But I didn't understand the question nor the answers there. And it looks like that even the answers and comments are contradicting each other.

set.seed(0)
df <- data.frame(a = rep(sample(1:4),10), b = rep(sample(1:4),10))
kripp.alpha(t(df))

This is the output.

 Krippendorff's alpha

 Subjects = 40 
   Raters = 2 
    alpha = 0.342 

How can I compute the confidence interval here?

Answer 1

You are right it is connected to bootstrapping. You could compute the confidence interval the following way:

 library(irr)
 library(boot)

 alpha.boot <- function(d,w) {
        data <- t(d[w,])
        kripp.alpha(data)$value
 }

 b <- boot(data = df, statistic = alpha.boot, R = 1000)
 b
 plot(b)
 boot.ci(b, type = "perc")

This is the output:

 Bootstrap Statistics :
      original      bias    std. error
 t1* 0.3416667 -0.01376158   0.1058123

 BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS
 Based on 1000 bootstrap replicates

 CALL : 
 boot.ci(boot.out = b, type = "perc")

 Intervals : 
 Level     Percentile     
 95%   ( 0.1116,  0.5240 )  
 Calculations and Intervals on Original Scale

there is also a R script from Zapf et al. 2016 look for Additional file 3 at the bottom of the page just before the references

Or you could use the kripp.boot function available on github MikeGruz/kripp.boot