SQLite transaction for CSV importing

Question

I\'m very new to python and have been working on my raspberry pi to get a script up and running to import millions of sensor data records into sqlite. I want to do this in t

Answer 1

The problem is that the object you get from the csv.reader method does not support the len() function. In fact, this CSV reader reads only when instructed to do so, and as such, it does not know how many records are in the file.

As such, you need to update the chunks method to deal with not knowing the amount of things to be chunked. Replace that function with this:

def chunks(data, n=10000):
    buffer = [None] * n
    idx = 0
    for record in data:
        buffer[idx] = record
        idx += 1
        if idx == n:
            yield buffer
            buffer = [None] * n
            idx = 0
    if idx > 0:
        yield buffer[:idx]

What this does is: it keeps retrieving records from your file as long as there are records to retrieve, and every n rows, emits a list of n records. For example:

>>> for c in chunks(range(10), 3):
...     print(c)
...
[0, 1, 2]
[3, 4, 5]
[6, 7, 8]
[9]