Read the last N lines of a CSV file in Python with numpy / pandas

Question

Is there a quick way to read the last N lines of a CSV file in Python, using numpy or pandas?

1. I cannot do skip_header

Answer 1

Option 1

You can read the entire file with numpy.genfromtxt, get it as a numpy array, and take the last N rows:

a = np.genfromtxt('filename', delimiter=',')
lastN = a[-N:]

Option 2

You can do a similar thing with the usual file reading:

with open('filename') as f:
    lastN = list(f)[-N:]

but this time you will get the list of last N lines, as strings.

Option 3 - without reading the entire file to memory

We use a list of at most N items to hold each iteration the last N lines:

lines = []
N = 10
with open('csv01.txt') as f:
    for line in f:
        lines.append(line)
        if len(lines) > 10:
            lines.pop(0)

A real csv requires a minor change:

import csv
...
with ...
    for line in csv.reader(f):
    ...