Splitting Rows in csv on several header rows

Question

I am very new to python, so please be gentle.

I have a .csv file, reported to me in this format, so I cannot do much about it:

ClientAccountID   Acc         


        

Answer 1

If I've understood your question correctly, you have a single CSV file which contains multiple tables. Tables are delimited by header rows which always begin with the string "ClientAccountID".

So the job is to read the CSV file into a list of lists-of-dictionaries. Each entry in the list corresponds to one of the tables in your CSV file.

Here's how I'd do it:

1. Break up the single CSV file with multiple tables into multiple files each with a single table. (These files could be in-memory.) Do this by looking for lines which start with "ClientAccountID".
2. Read each of these files into a list of dictionaries using a DictReader.

Here's some code to read the file into a list of StringIOs. (A StringIO is an in-memory file. It works by wrapping a string up into a file-like interface).

from csv import DictReader
from io import StringIO

stringios = []

with open('file.csv', 'r') as f:
    stringio = None
    for line in f:
        if line.startswith('ClientAccountID'):
            if stringio is not None:
                stringio.seek(0)
                stringios.append(stringio)
            stringio = StringIO()
        stringio.write(line)
        stringio.write("\n")
    stringio.seek(0)
    stringios.append(stringio)

If we encounter a line starting with 'ClientAccountID', we put the current StringIO into the list and start writing to a new one. When you've finished, remember to add the last one to the list too. Don't forget (as I did, in an earlier version of this answer) to rewind the StringIO after you've written to it using stringio.seek(0).

Now it's straightforward to loop over the StringIOs to get a table of dictionaries.

data = [list(DictReader(x, delimiter='\t')) for x in stringios]

For each file-like object in the list stringios, create a DictReader and read it into a list.

It's not too hard to modify this approach if your data is too big to fit into memory. Use generators instead of lists and do the processing line-by-line.