Set data type for specific column when using read_csv from pandas

Question

I have a large csv file (~10GB), with around 4000 columns. I know that most of data i will expect is int8, so i set:

pandas.read_csv(\'file.dat\', sep=\',\',         


        

Answer 1

Since you have no header, the column names are the integer order in which they occur, i.e. the first column is df[0]. To programmatically set the last column to be int32, you can read the first line of the file to get the width of the dataframe, then construct a dictionary of the integer types you want to use with the number of the columns as the keys.

import numpy as np
import pandas as pd

with open('file.dat') as fp:
    width = len(fp.readline().strip().split(','))
    dtypes = {i: np.int8 for i in range(width)}
    # update the last column's dtype
    dtypes[width-1] = np.int32

    # reset the read position of the file pointer
    fp.seek(0)
    df = pd.read_csv(fp, sep=',', engine='c', header=None, 
                     na_filter=False, dtype=dtypes, low_memory=False)