How to read a specific line number in a csv with pandas

I have a huge dataset and I am trying to read it line by line. For now, I am reading the dataset using pandas:

df = pd.read_csv("mydata.csv", sep =',', nrows = 1)

This function allows me to read only the first line, but how can I read the second, the third one and so on? (I would like to use pandas.)

EDIT: To make it more clear, I need to read one line at a time as the dataset is 20 GB and I cannot keep all the stuff in memory.

Looking in the pandas documentation, there is a parameter for read_csv function:

skiprows

If a list is assigned to this parameter it will skip the line indexed by the list:

skiprows = [0,1]

This will skip the first one and the second line. Thus a combination of nrow and skiprows allow to read each line in the dataset separately.

One way could be to read part by part of your file and store each part, for example:

df1 = pd.read_csv("mydata.csv", nrows=10000)

Here you will skip the first 10000 rows that you already read and stored in df1, and store the next 10000 rows in df2.

df2 = pd.read_csv("mydata.csv", skiprows=10000 nrows=10000)
dfn = pd.read_csv("mydata.csv", skiprows=(n-1)*10000, nrows=10000)

Maybe there is a way to introduce this idea into a for or while loop.

You are using nrows = 1, wich means "Number of rows of file to read. Useful for reading pieces of large files"

So you are telling it to read only the first row and stop.

You should just remove the argument to read all the csv file into a DataFrame and then go line by line.

See the documentation for more details on usage : https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html

来源：https://stackoverflow.com/questions/47586614/how-to-read-a-specific-line-number-in-a-csv-with-pandas

标签