Populate a dataframe from scattered files

Question

We have some data which is scattered in roughly the following format

• base dir
• • folder (for each entity)
• • <