Read .csv file from URL into Python 3.x - _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

Question

I\'ve been struggling with this simple problem for too long, so I thought I\'d ask for help. I am trying to read a list of journal articles from National Library of Medicine

Answer 1

Even though there is already an accepted answer, I thought I'd add to the body of knowledge by showing how I achieved something similar using the requests package (which is sometimes seen as an alternative to urlib.request).

The basis of using codecs.itercode() to solve the original problem is still the same as in the accepted answer.

import codecs
from contextlib import closing
import csv
import requests

url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv"

with closing(requests.get(url, stream=True)) as r:
    reader = csv.reader(codecs.iterdecode(r.iter_lines(), 'utf-8'))
    for row in reader:
        print row   

Here we also see the use of streaming provided through the requests package in order to avoid having to load the entire file over the network into memory first (which could take long if the file is large).

I thought it might be useful since it helped me, as I was using requests rather than urllib.request in Python 3.6.

Some of the ideas (e.g using closing()) are picked from this similar post