How to use python csv module for splitting double pipe delimited data

Question

I have got data which looks like:

"1234"||"abcd"||"a1s1"

I am trying to read and write using Python's csv reader and writer. As the csv module's delimiter is limited to single char, is there any way to retrieve data cleanly? I cannot afford to remove the empty columns as it is a massively huge data set to be processed in time bound manner. Any thoughts will be helpful.

Answer 1

The docs and experimentation prove that only single-character delimiters are allowed.

Since cvs.reader accepts any object that supports iterator protocol, you can use generator syntax to replace ||-s with |-s, and then feed this generator to the reader:

def read_this_funky_csv(source):
  # be sure to pass a source object that supports
  # iteration (e.g. a file object, or a list of csv text lines)
  return csv.reader((line.replace('||', '|') for line in source), delimiter='|')

This code is pretty effective since it operates on one CSV line at a time, provided your CSV source yields lines that do not exceed your available RAM :)

Answer 2

>>> import csv
>>> reader = csv.reader(['"1234"||"abcd"||"a1s1"'], delimiter='|')
>>> for row in reader:
...     assert not ''.join(row[1::2])
...     row = row[0::2]
...     print row
...
['1234', 'abcd', 'a1s1']
>>>

Answer 3

Unfortunately, delimiter is represented by a character in C. This means that it is impossible to have it be anything other than a single character in Python. The good news is that it is possible to ignore the values which are null:

reader = csv.reader(['"1234"||"abcd"||"a1s1"'], delimiter='|')
#iterate through the reader.
for x in reader:
    #you have to use a numeric range here to ensure that you eliminate the 
    #right things.
    for i in range(len(x)):
        #Odd indexes will be discarded.
        if i%2 == 0: x[i] #x[i] where i%2 == 0 represents the values you want.

There are other ways to accomplish this (a function could be written, for one), but this gives you the logic which is needed.

Answer 4

If your data literally looks like the example (the fields never contain '||' and are always quoted), and you can tolerate the quote marks, or are willing to slice them off later, just use .split

>>> '"1234"||"abcd"||"a1s1"'.split('||')
['"1234"', '"abcd"', '"a1s1"']
>>> list(s[1:-1] for s in '"1234"||"abcd"||"a1s1"'.split('||'))
['1234', 'abcd', 'a1s1']

csv is only needed if the delimiter is found within the fields, or to delete optional quotes around fields