Read a small random sample from a big CSV file into a Python data frame

Question

The CSV file that I want to read does not fit into main memory. How can I read a few (~10K) random lines of it and do some simple statistics on the selected data frame?

Answer 1

read the data file

import pandas as pd
df = pd.read_csv('data.csv', 'r')

First check the shape of df

df.shape()

create the small sample of 1000 raws from df

sample_data = df.sample(n=1000, replace='False')

#check the shape of sample_data

sample_data.shape()