Fastest way to read in 100,000 .dat.gz files

Question

I have a few hundred thousand very small .dat.gz files that I want to read into R in the most efficient way possible. I read in the file and then immediately ag

Answer 1

R has the ability to read gzipped files natively, using the gzfile function. See if this works.

rbindlist(lapply(dat.files, function(f) {
    read.delim(gzfile(f))
}))