Add “filename” column to table as multiple files are read and bound

Question

I have numerous csv files in multiple directories that I want to read into a R tribble or data.table. I use "list.files()" with the recursive argument set to TRUE to create a list of file names and paths, then use "lapply()" to read in multiple csv files, and then "bind_rows()" stick them all together:

filenames <- list.files(path, full.names = TRUE, pattern = fileptrn, recursive = TRUE)
tbl <- lapply(filenames, read_csv) %>% 
  bind_rows()

This approach works fine. However, I need to extract a substring from the each file name and add it as a column to the final table. I can get the substring I need with "str_extract()" like this:

sites <- str_extract(filenames, "[A-Z]{2}-[A-Za-z0-9]{3}")

I am stuck however on how to add the extracted substring as a column as lapply() runs through read_csv() for each file.

Answer 1

I generally use the following approach, based on dplyr/tidyr:

data = tibble(File = files) %>%
    extract(File, "Site", "([A-Z]{2}-[A-Za-z0-9]{3})", remove = FALSE) %>%
    mutate(Data = lapply(File, read_csv)) %>%
    unnest(Data) %>%
    select(-File)

Answer 2

You could use purrr::map2 here, which works similarly to mapply

filenames <- list.files(path, full.names = TRUE, pattern = fileptrn, recursive = TRUE)
sites <- str_extract(filenames, "[A-Z]{2}-[A-Za-z0-9]{3}")  # same length as filenames

library(purrr)
library(dplyr)
library(readr)
stopifnot(length(filenames)==length(sites))  # returns error if not the same length
ans <- map2(filenames, sites, ~read_csv(.x) %>% mutate(id = .y))  # .x is element in filenames, and .y is element in sites

The output of map2 is a list, similar to lapply

If you have a development version of purrr, you can use imap, which is a wrapper for map2 with an index

Answer 3

You just need to write your own function that reads the csv and adds the column you want, before combining them.

my_read_csv <- function(x) {
  out <- read_csv(x)
  site <- str_extract(x, "[A-Z]{2}-[A-Za-z0-9]{3}")
  cbind(Site=site, out)
}

filenames <- list.files(path, full.names = TRUE, pattern = fileptrn, recursive = TRUE)
tbl <- lapply(filenames, my_read_csv) %>% bind_rows()

Answer 4

You can build a filenames vector based on "sites" with the exact same length as tbl and then combine the two using cbind

### Get file names
filenames <- list.files(path, full.names = TRUE, pattern = fileptrn, recursive = TRUE)
sites <- str_extract(filenames, "[A-Z]{2}-[A-Za-z0-9]{3}")

### Get length of each csv
file_lengths <- unlist(lapply(lapply(filenames, read_csv), nrow))

### Repeat sites using lengths
file_names <- rep(sites,file_lengths))

### Create table
tbl <- lapply(filenames, read_csv) %>% 
  bind_rows()

### Combine file_names and tbl
tbl <- cbind(tbl, filename = file_names)

Answer 5

data.table approach:

If you name the list, then you can use this name to add to the data.table when binding the list together.

workflow

files <- list.files( whatever... )
#read the files from the list
l <- lapply( files, fread )
#names the list using the basename from `l`
# this also is the step to manipuly the filesnamaes to whatever you like
names(l) <- basename( l )
#bind the rows from the list togetgher, putting the filenames into the colum "id"
dt <- rbindlist( dt.list, idcol = "id" )