Getting “newline inside string” while reading the csv file in Python?

Question

I have this utils.py file in Django Architecture:

def range_data(ip):
    r = []
    f = open(os.path.join(settings.PROJECT_ROOT, \'static\', \'csv \', 
             


        

Answer 1

You must open your files as binary:

def range_data(ip):
    r = []
    f = open(os.path.join(settings.PROJECT_ROOT, 'static', 'csv ', 
                          'GeoIPCountryWhois.csv'), 'rb')
    for num,row in enumerate(csv.reader(f)):
        # Your things.

Note the 'rb' mode there; otherwise the file could be opened with native line endings, and the CSV reader doesn't handle the various forms very well. Certainly the copy of GeoIPCountryWhois.csv that I downloaded has clean \n line endings.

This is documented for the .reader() method:

If csvfile is a file object, it must be opened with the ‘b’ flag on platforms where that makes a difference.

If, however, your csv file is so corrupted as to still contain unexpected newline characters in unexpected places, use this file subclass instead as a stop-gap measure:

class CleanlinesFile(file):
    def next(self):
        line = super(CleanlinesFile, self).next()
        return line.replace('\r', '').replace('\n', '') + '\n'

This class guarantees there will be no newlines anywhere in the returned results except as the very last character (just the way the csv module wants it). Use it instead of the open call; the 'rb' mode modifier becomes optional in this case:

def range_data(ip):
    r = []
    f = CleanlinesFile(os.path.join(settings.PROJECT_ROOT, 'static', 'csv ', 
                          'GeoIPCountryWhois.csv'))
    for num,row in enumerate(csv.reader(f)):
        # Your things.