Inconsistent pandas read_csv dtype inference on mostly-integer string column in huge TSV file

Question

I have a tab separated file with a column that should be interpreted as a string, but many of the entries are integers. With small files read_csv correctly interprets the c

Answer 1

To avoid having Pandas infer your data type, provide a converters argument to read_csv:

converters : dict. optional

Dict of functions for converting values in certain columns. Keys can either be integers or column labels

For your file this would look like:

df2 = pd.read_csv('test', sep='\t', converters={'a':str})

My reading of the docs is that you do not need to specify converters for every column. Pandas should continue to infer the datatype of unspecified columns.