Recursively Edit CSV to Subdirectories using Pandas

Question

I have a series of subdirectory folders that each have a \"_Invoice.csv\".

/Invoice List/
              Invoice1folder/
                             ..._Invoice.         


        

Answer 1

This will do the job.

Instead of opening, removing columns, saving and moving on; I have opted for opening only with the reduced columns, saving this reduced DataFrame, then appending to df. This will result in all the reduced files being stacked in this one DataFrame.

Using path = "." goes from the current directory

from pathlib import Path
import pandas as pd


df = pd.DataFrame()
columns_to_keep = ['A','C']
path = "."
pattern = "*_Invoice.csv"

for file in Path(path).rglob(pattern):
    output_file = "{}/{}{}".format(file.parent, file.stem, "_Reduced.csv")
    _df = pd.read_csv(file, usecols=columns_to_keep)
    _df.to_csv(output_file, sep=",", index=False, header=True)
    df = pd.concat([df, _df])