Best practices for importing large CSV files

前端 未结 10 876
攒了一身酷
攒了一身酷 2020-12-14 16:39

My company gets a set of CSV files full of bank account info each month that I need to import into a database. Some of these files can be pretty big. For example, one is abo

10条回答
  •  北荒
    北荒 (楼主)
    2020-12-14 17:06

    I am reading a CSV file which has close to 1M records and 65 columns. Each 1000 record processed in PHP, there is one big fat MySQL statement that goes into the database. The writing takes no time at all. It's the parsing that does. The memory used to process this uncompressed 600MB file is about 12 MB.

提交回复
热议问题