Fast concatenation of thousands of files by columns

Question

I am using R to cbind about ~11000 files using:

dat <- do.call(\'bind_cols\',lapply(lfiles,read.delim))

which is unbelievably s

Answer 1

For fast reading of files, we can use fread from data.table and then rbind the list of data.table using rbindlist specifying the idcol=TRUE to provide a grouping variable to identify each of the datasets

library(data.table)
DT <- rbindlist(lapply(lfiles, fread), idcol=TRUE)